Fetching local JSON


#1

I'm trying to get at my local JSON file, which is very large, but all I'm finding are full stack solutions:

http://marmelab.com/blog/2015/03/25/fakerest-stub-for-rest-api.html

Is there any way to get around all this in a local Windows environment? I do have Node.js onboard, but lack the wherewithal.


#2

@mtf What are you able/allowed to do to get it? I've used the fetch() API before to get a JSON file and it worked pretty well for my situation. Usage for that would be something like this:

fetch(JSONUrl).then(function(response) {
  return response.json();
}).then(function(json) {
  // do a bunch of stuff
};

Support isn't great, but the modern desktop browsers seem to mostly support it: http://caniuse.com/#feat=fetch

If you're interested, David Walsh has a helpful blog post on fetch().


#3

This is what I am trying to work around...

will_json_data_query.html:22 XMLHttpRequest cannot load file:///D:/cc/discuss/brin/will_play_text.json. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.

The JSON and HTML are in the same directory. I can open it with Brackets but the file is so huge I cannot traverse it. I know I've worked with local JSON in the past but my brain is so fuzzy these days, nothing is coming back to me.

Day 6 of Wes Bos' JS Challenge30 gives us this code for reading in JSON:

const endpoint = 'will_play_text.json';

const plays = [];
fetch(endpoint)
  .then(blob => blob.json())
  .then(data => plays.push(...data));

which is throwing the above error.


#4

@mtf I don't know any ES6, and haven't gotten that far in the JavaScript30 course yet, so I'm not able to help you much with that code :confused:

Are you able to run a local server? That might fix the cros origin request it's upset about. Python lets you serve a directory from your terminal easily, if you have the module installed:

# python 2
$ python -m SimpleHTTPServer

# python 3
$ python3 -m http.server

The module comes preinstalled on some Linux distros, but I'm not sure if it comes on Windows.


#5

You'll love it when you get into it. I'm just scratching at the surface, myself.

This,

 .then(blob => blob.json())

is the equivalent of this,

.then(function(blob) {
    return blob.json();
})

ES6 offers implicit return if the curly braces are left out.

My Python 3 installation is,

C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\pythonw.exe

which I'm sure is not in the right place. A bull in a china shop is how I handled the install way back when and just never got around to setting this right. It is not configured so I don't think the command line will work. Think I need to take some time to get the install in the correct location and get my user environment configured. Right now I'm just happy to have IDLE.


#6

Launching python might be all you need

>>> import os
>>> os.chdir('/path/to/json/dir')
>>> import runpy
>>> runpy.run_module('http.server', run_name='__main__')
Serving HTTP on 0.0.0.0 port 8000 ...

(equivalent to python -m http.server in that dir)


#7

So far, so good. Thank you, @ionatan. Next question, can I use the Python server from within a JS script, or should I confine it to Python? Probably a dumb question. The other project that got me into this is the Friday Night one that @825orion posted in the CB. It's Python so it makes sense to stick to that environment, moving forward.

Time to learn how to work with JSON in Python...


#8

I thought python was just a way of making the file accessible without security complaints from .. something -- to make them available through http instead of from the file system

Can js use it? As long as it has the capability to launch other processes

Python has a json module so there's nothing difficult about that


#9

Seems my Notepad++ got hooped last night trying to open that huge JSON file and I may have to reinstall it.

Question: Assuming my working directory is set, would the URL be,

'http://localhost:8000/file.json'

?


#10

it'll be listed if you visit the root page in your browser


#11

And so it is. Thank you.

Directory listing for /

    insults_generator_romeo_juliet_825orion.py
    will_json_data_query.html
    will_play_text.json

Guess I'll need to install a favicon.ico file so I don't get this:

Serving HTTP on 0.0.0.0 port 8000 ...
127.0.0.1 - - [17/Dec/2016 16:16:34] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [17/Dec/2016 16:16:34] code 404, message File not found
127.0.0.1 - - [17/Dec/2016 16:16:34] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [17/Dec/2016 16:16:34] code 404, message File not found
127.0.0.1 - - [17/Dec/2016 16:16:34] "GET /favicon.ico HTTP/1.1" 404 -

Another question: Since the shell is busy with the server, can I open a second shell and have a command line while that is running?


#12

You got 200 OK for the page, let your browser ask all it wants for a favicon :rolling_eyes:


#13

I just dropped one in. Going to take some time to dig up docs on JSON and Python. Expect more questions... Thanks for your time.

One last question before I go... Would the Node.js server be configured to the same localhost port? Seems only right to use it if I'm working with JS (down the road)?


#14

Got my answer as far as using that server with JS.

Uncaught (in promise) TypeError: Failed to fetch
will_json_data_query.html:1 Fetch API cannot load http://localhost:8000/will_play_text.json. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Looking up 'request mode'

This is the output from the server in the shell:

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 54142)
Traceback (most recent call last):
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 318, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 344, in process_request
    self.finish_request(request, client_address)
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 357, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 684, in __init__
    self.handle()
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 415, in handle
    self.handle_one_request()
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 403, in handle_one_request
    method()
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 631, in do_GET
    self.copyfile(f, self.wfile)
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 792, in copyfile
    shutil.copyfileobj(source, outputfile)
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\shutil.py", line 76, in copyfileobj
    fdst.write(buf)
  File "C:\Users\Mr\AppData\Local\Programs\Python\Python35-32\lib\socket.py", line 593, in write
    return self._sock.send(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
----------------------------------------

#15

What is running your js code, and where is it getting the js code from?


#16

My JS code is above, running as HTML <script></script> in Chrome.

const endpoint = 'http://localhost:8000/will_play_text.json';

const plays = [];
fetch(endpoint)
  .then(blob => blob.json())
  .then(data => plays.push(...data));

An existing connection was forcibly closed by the remote host

Does this mean there is a restriction in the JSON file header? Or is a browser thing?


#17

Serve the script with the python server? Wouldn't it be the same origin then?


#18

Exception happened during processing of request from ('127.0.0.1', 54142)

This has me confused. It looks like port 54142 is the connection? D'oh! It's the port the request is going out on. Still confused.

Think I best stick to one language and stop mucking about. Going to put this JS convo on hold and concentrate on finding the Python JSON docs. It'll be time better spent, and I won't be wasting yours. Thanks again!


#19

If you can open the json directly in your browser then it's probably not got to do with python


#20

Uhm. Yeah, if it's local then.. js is probably not the tool for the job