Transcription
That there. Okay, so welcome everybody. Here's the agenda for today. We're going to go through quickly some overview materials with happy, what it is, and how to use it, and some of the things that have grown up around it. And then try to get to a Q&A time where people can talk about their their own experiences or questions. And then we'll just be around for for the discussion.
So this is geared towards people who don't know too much about happy yet, so there will be some, uh, just very basic stuff. So go ahead and get started with that. Uh, can you see this okay? I think I'll just leave it like the, uh, not switch to presenter mode. Is this visible?
[Music]
Yes, okay. All right. So what is happy? Well, it stands for the Heliophysics Application Programmers Interface, meaning it's, and it's just a standard interface for serving time series data. Something you put on a server, and then clients can talk to it in a standard way. And in terms of where do we fit in the, in the NASA science scheme? You know, there's different Earth science, uh, different divisions, astrophysics, are in science, heliophysics, so forth. And then within heliophysics, there's three different regimes. And the focus so far has been magnetospheric and heliospheric, but also aeronomy has lots of time series data too. So time series data just means primarily indexed by time, and then you have multiple columns, either of scalars or arrays or multi-dimensional data. So it was designed with this, uh, science scheme in mind. However, it really is a generic interface. There's nothing heliophysics specific about it. We did have to decide on a representation for time, so we went with an ISO standard, and you can use either the day of year or year, month, day arrangements of that. So I know some fields use different types of values, different representations, but at least this one is a standard.
Okay, so why, what's the motivation for happy? Well, what we realized was, you know, a lot of the file formats had really converged to kind of a set of standards, but the interface is getting to services had not standardized. So each different institution, Goddard or LASP or University of Iowa, different places, the Planetary Data System, they each have their own interface to getting to data. And that then requires unique software to reach each particular service, and so that's not too interoperable. So what we're, what we thought of is, well, if we could have services add on, not replace, but just add on a simple interface, just to get to the time series data, then you could use common software, and that's much more interoperable. So it's not anything complicated. It's just mostly getting everybody to to agree on what that lowest common denominator should be. And so there is, uh, we have been able to get a lot of agreement. Here's some of the different institutions that are either having happy servers up and going now or planning on them. And, uh, so it's quite, you know, it's catching on, and we think it's working. So I don't need to dwell on that too much.
We are COSPAR recommended standard. And here's the just the brief blurb that they published on this, and they're referencing our version 3.0 of the specification. So it was recognized in that way as well.
So how does it work? Well, it's a specification. So that's a little bit abstract. What does that mean? It just means that it's telling a server how it should behave in two key areas: what's the request interface and what's a response format. The request is just very simple URLs. It's called a RESTful interface for Representation, Representational State Transfer. And then the response format is also very simple. The server must provide a comma-separated value response where the data is streamed, and there are faster formats supported, JSON, and also a binary response. And so it's quite easy to write a parser for that stream that comes back. And the stream means that file boundaries, anything underneath the data, how it's stored, is no longer visible or even relevant, just getting that happy stream back.
So each happy server must support these now five endpoints. And so whatever your server is, you have to have a slash happy piece. And then the different endpoints. The about endpoint just gives you some contact info on how to cite this server. Capabilities, uh, you must implement comma-separated values, but you can also indicate in the capabilities if you, if you support other formats. The catalog, now we're starting to get into the details, that provides a list of the data sets. And then the info, you provide a data set ID, and it gives you metadata about a particular data set. And then, uh, the data endpoint is where you, it will stream back the numbers, CSV by default, but you can also optionally support JSON or binary.
So just a quick example. This is, uh, from an info request for a particular data set. This is truncated just to kind of show you roughly what it would be. Usually there's more parameters. This just has two parameters. The first one must always be a time parameter, and then after that, you have data parameters. And there's enough information, um, in the metadata that you can interpret this properly and and plot it.
And then, John, to clarify, I think you shot your slide. The request normally would have info and then a question mark and then the ID of the parameter that's requesting the info about. That's right. Yes. Thanks. So, yeah, and the ID is not mirrored back in the response. Once you start mirroring back response information from users, you start getting into more security issues. So the response just spills out the metadata and not the data set ID. So, okay, so that's basically it for the intro, motivation. And we thought we would jump now to some demonstrations. And, uh, I'll hand that over to Sandy.
So the, uh, usual question people have is, well, how can I use this? So we made a very quick demo run in order to show it being used from a variety of interfaces. A happy demo. Happy is a specification and protocol, but also includes tool sets. And this is a look at the different ways that you can fetch happy data according to the environment that you want, whether it's a web fetch through Python, PySPEDAS, IDL, happy-server.org capabilities, including script generation, and external tools such as Autoplot.
We'll start with a very basic web fetch. You give a URL, you include what is the data set that you want, what is the time range. By default, it will get all of the data for you and return it as CSV format, that being the default. So looking at the data, you have CSV format as specified by the happy server. If you want just a single data item or a subset or a list of data items, in this case, we'll get just the SIM H variable, then we add that in as a parameter, and we get just the variables that we requested along with the timestamp. Timestamp plus variables is the happy format.
Working with this in Python, you install the happy client and optionally the happy plot packages, or use the plotting package of your choice. And again, you're going to set the server, data set, and parameters for what you wish to analyze. Looking at in a Jupyter notebook, we import the packages, we set our server, see the web, again, the data set, the specific parameters that we're fetching, in this case, BF1 and BF1 log. We do our fetch, and then we call happy plot and just throw the data at it in order to plot. Once the data is there, it is fetched, and the metadata includes what this actually means. If you're like, well, what is BF1? Magnetic field magnitude at one-minute increments. What's BF1 log? Same thing, log scale. The metadata tells you what the data is that you have fetched.
In addition to being able to do line plots, you can also fetch spectral data. Spectral data is just another form of time series. Um, give this a second to plot because it is a notebook, and there you have some MMS spectra plotted for you.
If you work in PySPEDAS, it's the same thing. You install the PySPEDAS package, and the latest version has support for fetching data from happy compliant servers. So we're good with having installed the package. We import again. We set a server, see the web, the data set, the specific parameter that we're getting, and a time range. It fetches the data, and here we're using tplot in order to plot it, and there's our data. And again, loading in some MMS data and ignoring the warns, we again get our time series or our spectrum plotted as we see fit.
IDL. If you work in IDL, surprise, surprise. You install the package, you set the server, data set, parameter, start, and stop. You call the happy protocol, and you are good to go. So here's data. There's the metadata around it. You can print it, you can plot it, you can work with it in IDL as you wish. Okay, we'll print in a moment, and there you have your data.
There, happy-server.org has a list of all servers that are compliant. For example, see the web. When you choose see the web or any of the items from the dropdown, it will list for you which data sets are available. In this case, we're going to go with the same data set that we've been using for this demo, specifically parameters. It will tell you which parameters, okay, data items are available. And as before, we'll set that, we'll set a time range. And here you have three choices: you can get the data, you can get an image, or you can get a script. So if you choose the data, in this case, CSV, there you have it with a link for downloading the data. Instead of the data, if you wish to quickly plot it, then you get your plot of the data. And particularly helpful if you need a script, then you can get IDL, MATLAB, Python, or Autoplot generated. So if you want to know how to fetch your data in MATLAB, this will generate the code for you. So as a code generator, this is excellent for putting into your workflow. It will generate Python code, IDL, and we'll also make an Autoplot file in order to use it. And again, we're going with the same thing: server, data set, parameters, start, stop. That is the happy specification.
Again, external tools like Autoplot will gather data for you. So you call up Autoplot and tell it what you want. So we're going to look for happy feeds, and here we have see the web loaded up. And again, I just have to scroll down to the data set that I wish to grab from. We'll grab that one. Items, it lists some suggested items. We're going to grab the magnetic field information. Ignore the distance. We call it the plot, and Autoplot will generally take whatever you throw at it and try to plot it intelligently. You can then examine the metadata or work the entire project further.
So that concludes our happy demo of how to access happy data in a variety of subsystems. And just add in, a big fan of the script generator, and we're adding SPEDAS and PySPEDAS script generation as well, based on feedback. Feedback is super important. If you want something, let us know.
Okay, thanks, Sandy. And then Bob is going to show some of the showcase some of the items we have at our GitHub site, some of the different projects. There's a few of them that are particularly helpful if you're trying to design a server or work on your own clients.
So we can't hear you yet, Bob. You might... Yes, here I am. Sorry about that. No, no problem. Okay. Uh, so this is a brief presentation on some of the happy ecosystem. We, happy is primarily a specification. This essentially defines how a server is going to respond to a query for a data set, parameter, and time range, uh, what the output formats will be, and, you know, how it responds to errors and so on. And that's all part of the specification that's well developed. And as John said, it was developed based on, you know, collaboration with of people who are essentially developing servers with similar capabilities, and we sort of took the best of everything and put it together for the specification. Of course, we all know it's easy to create a specification. It's, uh, comparatively easy to create a specification. Uh, one of the challenges is to get people to use it. And so we've put a lot of effort into developing software that uses the spec. And Sandy showed you some of the tools that already exist out there.
If you're a developer, I recommend that you start with the happy repository. So it's github.com/happy-server. And I try to keep this list of repositories. We have 22 repositories now. I try to keep it up to date and to point to all of the software that exists out there that uses happy in one way or the other. And so I will just briefly go through some of these repositories and what they do.
The happy client. There's a Python client, which is, it's a pretty simple library. I mean, the guts of the code is is maybe a couple of hundred lines. It ends up being maybe 800 because we've had a lot of extra stuff that's not totally necessary in there. But all it does is is fetch the data and puts it into a Python, or excuse me, a NumPy, a NumPy array. And that's where it stops. And then there are other libraries that we expect to build upon that. Um, and so all of that's open source and it's installable from pip.
Python SPEDAS. This is just a link to where to get started with PySPEDAS and how to use it. Uh, if you're developing a server, you've probably seen the typical server landing page, which just sort of lists says, this is a happy server, here are the endpoints that John talked about. This project is essentially, it has two parts. One is just a very simple web page, a plain old web page that's, um, that looks like this. Oops, that's the wrong link. Um, I don't have the, I missed clicked the link. Anyway, so it just has a very simple landing page, and then the second piece of that is the code that powers, powers this. So if you have, if you've developed a happy server and you just want a a little bit of a fancier interface, you can download that server UI code, and then this server list will just be, well, there won't be a server list, it'll be just a list of data sets from your server.
Yeah, and so we, we really make an effort to try and get, you know, people to contribute code or to, you know, inform us about what exists and what's out there. Um, and so there are two servers that you can start with if you're developing a happy server and you want, you don't want to do all of the coding yourself based on inspection of the documentation. So there's one in Node.js, there's one in Python, and I think there's one in Java. Pieces of a Java one floating around there, and we'll add that to the list.
Um, another piece of software that was developed was a verifier. And so essentially, after you've developed a server, you want to make sure that it's compliant with the standard. And so what you can do is just go to happy-server.org/verify, and you can plug in a, so here I have a test data server. And essentially, you just pass the URL to your server or wherever it is, and this web interface will start hitting your server with various requests and doing various checks to make sure that you're implementing this spec properly. It checks the metadata, it also makes bad requests and make sure, make sure that the response is as expected. It'll also attempt to hit your server and try to create a plot. That's that's a good way to catch errors in your data. And so this, this has been a pretty useful tool for people developing the spec because the spec is, it's, it's simple in principle, but of course, there are many, many gotchas with any specification.
There's a separate Python plotting library that's, uh, standalone. It's, it's pretty basic, and we typically recommend that, you know, this plotting Python, that Python was just developed for essentially verification, and there are other Python libraries like the, uh, PySPEDAS, that is more suited for stack plots and other more complicated things. A bunch of clients. There's a new client that's in R that's partially developed that'll help you get started, a MATLAB one that's pretty well developed, and then various other tools out there. Yeah, so this is sort of the starting point that if you're developing happy, you want to look to see what exists here. And we always encourage people that, you know, if if you have modifications or especially bug reports, that you should report them. And then the code generation. There's a comment that says, if you encounter any issues, feel free to submit an issue. And, yeah, so that's, that's essentially what exists out there. You know, you should probably inspect this, especially if you're starting a server, and look at existing servers that are out there just to get up and running.
The server Node.js essentially is was written so that if you have something that can generate data from the command line, then you can install the software and it'll do the rest. It'll do the web interface and the checking and the validation, and the job on your end is essentially to produce, uh, a happy formatted stream. So that's, uh, a very brief overview of this.
And Bob, let me add to that. Um, so there's also a list of of known valid servers that we keep, and we are kind of constantly monitoring them. And as they add new data sets, we, uh, kind of register those data sets. Um, and we are, like Autoplot, when it starts up, it'll, it'll grab that list of of known servers and provide that to the the user of Autoplot for known things. So anyway, as you as you ramp up a server and as you feel it's ready for the world to start using, um, let us know when we can add it to this list.
Is that a direct email to you or, um, who should we let know? Uh, maybe just post an issue. Okay. Um, yeah, or or a pull request, or John, if you're working with him. Um, right. Yeah, you can, you can contact us. But yeah, if you want to, um, just post a pull request, add your information in there and say, hey, here's mine, and then we'll take a look at it. Probably run it through the verifier. Yeah, because that's a very comprehensive way of really hammering a server to see, does everything flesh out right? Yeah. Typically, our RV-SPICE data has been online on our server for probably a year now. So, okay, nice to get that available. Yeah, usually what we do is we'll add it to the, um, we'll add it to the dev, and then and then it'll be visible. The dev stuff is visible through servers-dev. So if you don't want to advertise it to the world, it'll show up in this list of things that, you know, aren't aren't quite ready for prime time, but are useful enough to be tested. And so that's another, another pathway you can take is to go through the, um, you're doing a pull request for the dev, dev list, great. So you have a RB-SPICE, RB-SPICE data is online at the Fundamental Technologies happy server, happy.ftx.com. Oh, right. I think we have that. Uh, there we go. Yeah, I think I moved it to the dev, uh, temporarily. I'll talk to you about that, but thanks.
Okay, well, um, I think that's it on the formal stuff. Just wanted to run through an overview of what's out there, how to use it, and some of the other things that are sort of in support of various happy efforts. So let me grab the screen back here. So yeah, just another shout out to the verifier. If you are developing your own server, this is a great way to check it. And you can either have the website check it, or you can download the verification code yourself and just run it locally. You don't have to, uh, you don't have to send anything out over the web until you're ready.
So I think we'll just jump now to Q&A. Uh, I guess we have some questions already, um, in the chat. Yes, I saw. I tried to look at those, but then I realized I was going to lose the the, uh, discussion thread if I paid too much attention to the chat. So I'll switch to that. And I was going to go ahead and stop the, uh, recording now. That way there's no, you know, people make sure to feel free to, you know, ask any question they want. So I'm going to go ahead and stop that.