NASA Open API for APOD

[RJN]

This thread is to discuss the open API that NASA created for APOD a few years ago. To find this go to https://api.nasa.gov/, scroll down and look for the APOD entry.

It has come to my attention that this API has some flaws. It is my hope that this Asterisk thread will illuminate these flaws and start the process of fixing them. This has come to my attention (again) recently because there are at least two recently-started, volunteer-run, APOD-related mirrors that are using the NASA Open API for APOD and emailing me. I think several others are also using the API to ingest APOD content. I would like to help but I am not sure how, at present. I will be requesting that the people who email me about the API alsopost on this thread so that commonalities can be seen, needs established, and, hopefully, solutions found.

- RJN

[Ivan]

Hi everyone. So, the issue I have come across recently is that API doesn't provide 'copyright' attribute (among others, as stated by @LeMorrow in this GitHub issue) in JSON-response for any APOD entry that has more than one person/organization listed in this field on the webpage.

For some reason, it was only today that I realized APOD-API is actually on GitHub, so I will take a closer look at the code tomorrow, but I believe this is issue should not be too difficult to fix.

[videotizer]

Hi @Ivan,

Just to be clear, you mean 'copyright' attribute is missing form the README.md file, right? Because there is no issue with the actual returned JSON object; it does include the 'copyright' key provided it exists for the date. If that's what you meant, then, and as you mentioned, @LeMorrow has created a pull request to address the GitHub issue you mentioned.


Hi @RJN,

Thanks for creating this thread to discuss APOD API. If I may suggest, It would be more efficient to use the GitHub repository located at https://github.com/nasa/apod-api to address technical issues.

[Ivan]

it does include the 'copyright' key provided it exists for the date

That's true, however what I meant was not the README issue.

You see, I'm running APOD's image on Telegram and @RJN pointed out that I do not mention who the credits go to, although this info is always on the APOD's webpage. I guess the info just isn't passed to the API, when there is more than one person mentioned in the 'copyright' section on the webpage, thus 'copyright' attribute doesn't exist for this date's entry.

I'm not saying API isn't working properly, but if we want the developers to be able to mention copyright info on their mirrors of APOD, we should figure out why this data is not passed to the API in the first place. Maybe you could explain that, since I am a bit confused by this whole situation.

P.S. To work it out faster and not spam into this thread would you like to contact via email or, maybe, Keybase?

[RJN]

Since several people are following this thread, and since others may benefit from it in the future, I appreciate any useful discussion or correspondence that can be contributed to this thread. Even if you describe mistakes, you may be saving others in the future from following those same mistakes. Thanks if you can. - RJN

[PawelPleskaczynski]

Hello, I'm the creator of Node.js version of the APOD API, the source is available here: https://github.com/PawelPleskaczynski/apod_api

While working on it, I fixed problems that the official API encounters, like not always returning proper copyright field or crashing on list request when at least one day would 404.

Also, I added some new functionality, like search endpoint or request caching so the user can be served much quicker.

Here's the main "workhorse" that scapes the webpages: https://github.com/PawelPleskaczynski/a ... /loader.js

I focused on functionality and reliability, not elegance of code so the loader.js file I attached is quite long and maybe repetitive, surely it's not elegant, that's all because I noticed that APOD website changed three times and in these "eras" I had to scrape the website differently.

I'll be happy to help with issues of the official API, so please ask!

By the way, would it be possible to batch modify all the APOD websites to include proper id tags? It'd be much easier to differentiate from eg. copyright text and description text

[videotizer]

@Ivan

I guess the info just isn't passed to the API, when there is more than one person mentioned in the 'copyright' section on the webpage, thus 'copyright' attribute doesn't exist for this date's entry.

Can you be more specific and give an example such as a date where you had issues with.

P.S. To work it out faster and not spam into this thread would you like to contact via email or, maybe, Keybase?

I don't mind emails, but as @RJN said, it's better to keep this conversation public so that "others may benefit from it in the future".

[videotizer]

Hi @PawelPleskaczynski

Thank you for your efforts on creating a Node.js version of APOD API. No one here judging any "elegance of code", all is needed is that it does the job. But I believe the best approach is to contribute to the official apod-api on GitHub, and then create a Node.js wrapper.

[PawelPleskaczynski]

Hello @videotizer,
It's not a wrapper of the official API, it's my own implementation, written from scratch. I posted it here so people can look at my code, at official API's code and compare them, seeing what can be improved (in both). For now, I know that scraping logic needs to be rewritten in the official one to include these different APOD website "eras"

[videotizer]

Hi @PawelPleskaczynski

I know it's not a wrapper, and totally appreciate that you did your own implementation. All I'm suggesting is to focus our efforts into the official API so that it can serve the maximum number of API users, regardless of their language or environment. If we have a solid official API, it would make it easier and simpler to consume the data without having to worry about creating different implementations to cover missing features or to fix issues.

As I said, I appreciate what you've done, and i also appreciate that you already contributed to APOD API on GitHub, and we, as developers, should follow suit.

[PawelPleskaczynski]

@videotizer, okay, I see, I totally agree with you.

So what should be done first? I think that the official API repo should be connected to the Heroku pipeline. That way, it will be easier to maintain and the API will be up to date with the code. Also, it would be great if PRs were reviewed (and possibly merged) more often by maintainers. I know that this API is not the highest priority there but still, we can't forget that it's used by many people

By the way, are there any plans to remove the API key? I don't think it's very necessary

[RJN]

The API was created for APOD by others at NASA not directly affiliated with APOD. I am not currently in touch with them -- almost everything I know is on the NASA API page. So I don't know if there are plans to update the API key. If it is important, I can ask. - RJN

[PawelPleskaczynski]

No, it's not that important, I was just curious, since (as far as I know) Heroku doesn't have connection limits, even on a free plan, so I thought API key is unnecessary.

The more important thing for now would be to connect the GitHub repo to the Heroku pipeline, so online API would stay up to date with the code on GitHub.

[RJN]

I am sorry but I am not familiar with Heroku pipelines. Can you give an explicit list of things that it would be good to do for the NASA APOD API, and why?

[PawelPleskaczynski]

Sure! I'm not an expert, so some of my insights may be wrong, but here's what I think would be good to do:

- connect GitHub repository to Heroku so the online version of the API would be up to date with the code on GitHub. For example, I can see that my fixes are not present in the online version (eg. this URL crashes: https://api.nasa.gov/planetary/apod?api ... 2018-10-07 even though my fix was merged: https://github.com/nasa/apod-api/pull/11). Maybe make a separate branch with stable version that gets pushed to Heroku (and if new version is stable, maintainers would merge code from master)?

- maybe add some new maintainers so the repository will be maintained more actively (the last commit was 8 months ago)

- remove the API key because I think it's a bit pointless (if it gets removed, some users might still use their own keys in the URLs, so these keys would be ignored)

- change the way the API scrapes the data, because sometimes some fields aren't returned on some dates and most often it's the copyright field that isn't returned, like here: https://api.nasa.gov/planetary/apod?api ... 2019-11-03 vs here: https://apodapi.herokuapp.com/api/?date=2019-11-03

- it's just a suggestion, but the official API on GitHub could be replaced with some other implementation, like my own, because, in my case, it's faster, scrapes the data more reliably, uses cache and has more functions

By the way, Mr. Nemiroff, do you have control over old APOD pages on the website? Would it be possible to batch modify these pages to add HTML tags to some elements, like description or copyrights fields? Scraping and parsing pages would be much easier then.

[videotizer]

Thank you @PawelPleskaczynski for all the details you provided and for giving an example of where the copyright data is missing.

Here's my take:

• Unify the API URL; having 2 is confusing and would cause many maintainability issues, which is probably why your fix was not reflected in the primary API URL.
• Fully agree about the repository's maintainers; we all know that maintaining a repo is time consuming and requires devs with enough time on their hand to handle the task.
• Personally, I don't mind the API key if it serves getting some statistics on usage that could be helpful in improving the service.
• If the above are not possible, in particular unifying the URL and having the time to maintain it, then, personally, I wouldn't mind the idea of rewriting the whole thing if it makes it more stable and efficient; of course such decision needs to be properly discussed and weighted and I would leave it as a last resort.

[RJN]

By the way, Mr. Nemiroff, do you have control over old APOD pages on the website? Would it be possible to batch modify these pages to add HTML tags to some elements, like description or copyrights fields? Scraping and parsing pages would be much easier then.

I do have control over old APOD pages, but I am notoriously reluctant to modify them. There has been recent discussion (elsewhere) of doing just that to bring these old APOD pages into HTML5 compliance, possibly also by adding companion CSS files. They were created for HTML3, by the way. What I might be willing to do is to keep those pages the same but allow a program to ingest them and create a parallel archive that is HTML5 compliant -- and now contain useful description and copyright fields. However, my programming skills for this type of thing are rusty and most of my time is spent doing my day job(s). Still, if someone wants to take up this task, I would be happy to let them, and of course credit them where appropriate.

- RJN

[videotizer]

Still, if someone wants to take up this task, I would be happy to let them, and of course credit them where appropriate.

I’ll be more than happy to contribute as needed.

How is the code maintained? Any code repository? If not, I suggest to start by open sourcing the code and hosting on GitHub or GitLab, so that contributing becomes easier and transparent. BTW, GitHub recently announced their Arctic Code Vault that will be used for storing public repositories hosted on their platform for 1000 years. The snapshot will be taken on 02/02/2020, so, if we have everything setup before that date, will ensure APOD is going to be kept for a thousand years! What a fascinating idea!

[PawelPleskaczynski]

@RJN,
I think I got a better idea than modifying these old pages. I thought it'd be possible to rewrite whole APOD website using the latest technologies. First of all, full backup of every APOD page would need to be made, then, a database (like MongoDB) would be created and populated with every APOD to date. It'd be done using eg. a modified version of my API (though it's not perfect and still, some days would need to be added/fixed manually). Then there would be a backend written in Node and frontend written with help of some framework (I like to work with Vue.js) and also an admin dashboard for you to add new APODs. After all of this is done, the possibilities would be endless. It'd be possible to add new features easily, like list view of APODs, search function, etc, and also code of the frontend could be open-sourced since it doesn't allow to modify the database, just read from it. Speaking of reading and writing to the database, we have two options: either connect directly to it on the frontend or connect from backend, allowing for easy creation of an API. The only drawback I can see is that some programs and APIs would be broken because they couldn't find and parse the classic pages, so I thought it could be resolved with a new path to new pages? So for example we would have two paths for the same entry: https://apod.nasa.gov/apod/ap191126.html for the legacy page (unchanged address) and https://apod.nasa.gov/apod/new/ap191126.html or something for the new page. While writing this I noticed that you'd have 2x amount of work so maybe the database of the new website would fetch the new APOD from the classic website. Please let me know what do you think about this!

[RJN]

Thanks Pawel, for your work, thoughts, and suggestions. You are right that many digital venues now ingest old APODs in their present format, and so if the format changed many of them would break. Also, I would like APOD base HTML code and usage software to be as ridiculously simple as possible. APOD's old HTML3 still works nearly 25 years after it was first written. I would like to think one reason for this is because it was so bare-bones basic and ground-level simple. It would be nice for any new solution to be so basic and simple as well so as to last another 25 years. Also, I don't see how this proposed upgrade would create 2x the amount of work for Jerry and me (and other potential APOD writers), but 2x would not be acceptable. Ideally, the new solution would have us take the same or even less time in non-writing mode than before.

I like the idea of keeping the legacy pages. People, codes, and APIs could still access the legacy pages. Or, if they want, the new improved HTML5 compliant pages that have easier-to-parse credit and copyright information.

[RJN]

I've thought a bit more about this. Can anyone suggest minimal changes to APOD's current HTML3 template that would

- bring it up to HTML5 compliance
- make it easier for APIs to scrape copyright information
- not break most programs that ingest APOD's current HTML3
- cite a CSS file that can make APODs look better but would not be essential -- the HTML5 should be able to work minimally just by itself

My naive hope is to just update and/or change out a few lines of the existing HTML3. Thanks to anyone who may attempt this.

- RJN

[geckzilla]

- not break most programs that ingest APOD's current HTML3

No one knows how most "programs" ingest APOD's current HTML. Each individual knows how they did their own, but there are multiple ways to parse it, and it's impossible to know how each did it without asking them individually, or acquiring their code.

FWIW I think there is a fair chance that what I've already done for you does all of that, though.

[RJN]

No one knows how most "programs" ingest APOD's current HTML. Each individual knows how they did their own, but there are multiple ways to parse it, and it's impossible to know how each did it without asking them individually, or acquiring their code.

Good point. So then ignore that request.

[RJN]

OK so now after some email discussion with Jerry I have created an APOD page that

- is HTML5 compliant
- has minimal changes from the old HTML3 code
- uses a CSS file (APODstyle.css) but will work without it

That file is here: https://apod.nasa.gov/apod/fap/pagetest5.html . Please have a look. Criticisms, including called me a Luddite, are welcome.

This still does not make it easier for APIs to scrape copyright information, which we would still like to do.

[videotizer]

I suggest the following:

• add this CSS class ".center { margin: 0 auto; text-align: center; }" so that it can be used for any centering needed without needing to add it as inline style
• lower case for all elements' tag names and properties such as the "img" tag
• add "img { max-width: 100% }" so that you don't have to add it as inline style
• <meta charset="UTF-8"> should be the first element in head
• add <meta name="viewport" content="width=device-width,initial-scale=1"> to head
• group different page sections semantically such as header, main, footer
• add ids to elements where appropriate - this makes it easier to scrap

I did a revised version that includes the above suggestions and also includes the CSS set in a <style> tag (separate CSS file optional). It also relies on using HTML5 semantic elements and elements ids to make easier to scrap the data when needed. Source code can be found at GitHub https://gist.github.com/videotizer/0493 ... e257678724. Please have a look and let me know if you need any additional help.

I also would like to remind you of what I previously proposed to host the code on a platform such as GitHub or GitLab, so that contributing becomes easier.