Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OMNI (not OMNI2) Data Not Up to Date #731

Open
julia-claxton opened this issue Feb 22, 2024 · 8 comments
Open

OMNI (not OMNI2) Data Not Up to Date #731

julia-claxton opened this issue Feb 22, 2024 · 8 comments

Comments

@julia-claxton
Copy link

Hello! I have noticed that my OMNI data only goes through January 2022, no matter how often I update it. My OMNI2 data is up to date, however using OMNI2 data in calculation of L-shell (my use case for the OMNI data) results in an error saying that the OMNI2 data does not have Kp index. MWE is given below.

Minimal example to reproduce issue:

import spacepy.toolbox, spacepy.omni

spacepy.toolbox.update()
print("OMNI Data Range: ", end = "")
print(spacepy.omni.omnirange())


import datetime
import spacepy.coordinates, spacepy.irbempy

time = spacepy.time.Ticktock(datetime.datetime(2022, 5, 1, 0, 0, 0), 'UTC') # Arbitrarily chosen
position = spacepy.coordinates.Coords([2, 2, 2], 'GEI', 'car', units = ['Re', 'Re', 'Re'], use_irbem = False) # Arbitrarily chosen
spacepy.irbempy.get_Lm(time, position, [90]) # This line throws the 'data out of range' error

Error message/Traceback:

OMNI Data Range: (datetime.datetime(1963, 11, 27, 14, 0), datetime.datetime(2022, 1, 19, 23, 0))

Traceback (most recent call last):
  File "/Users/<username>/Downloads/mwe.py", line 12, in <module>
    spacepy.irbempy.get_Lm(time, position, [90])
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1467, in get_Lm
    results = get_Lstar(ticks, loci, alpha, extMag=extMag, options=opts, omnivals=omnivals)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1663, in get_Lstar
    DALL = _get_Lstar(ticks, loci, alpha, extMag, options, omnivals, landi2lstar)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1367, in _get_Lstar
    d = prep_irbem(ticks, loci, alpha, extMag, options, omnivals)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/irbempy/irbempy.py", line 1722, in prep_irbem
    omnivals = omni.get_omni(ticks)
  File "/Users/<username>/Library/Python/3.9/lib/python/site-packages/spacepy/omni.py", line 257, in get_omni
    raise ValueError('Requested dates are outside data range')
ValueError: Requested dates are outside data range

OS, Python version, and dependency version information:

macOS-13.4.1-arm64-arm-64bit
sys.version_info(major=3, minor=9, micro=18, releaselevel='final', serial=0)
numpy=1.26.1
scipy=1.10.1
matplotlib=3.8.0

I am using a MacBook Pro with an M1 (Apple silicon) processor.

Version of SpacePy

spacepy=0.4.1

Downloaded via pip and updated prior to running MWE script. I am running spacepy with Python 3.9 because that is the only way I could get spacepy to run properly on my machine.
@candeynasa
Copy link

VIRBO.org is no longer getting updates. OMNI data are retrievable from https://spdf.gsfc.nasa.gov/pub/data/omni/omni_cdaweb/ (CDF files) or via the CDAS and HAPI APIs from CDAWeb.

@julia-claxton
Copy link
Author

I think it would be smart to have spacepy pull from that website if VIRBO is now out-of-date, since it will just get more out-of-date the more time goes on.

Either way, I'm not sure how to get spacepy to work with the provided CDFs for my purposes. The directory that L-shell calculations pull OMNI data from (/Users/<username>/.spacepy/data/qindenton_daily_files) are all .txt files on my machine, so .cdf won't work there. I have tried getting spacepy.irbempy.get_Lm() to use a more up-to-date database (e.g. OMNI2hourly), but that throws KeyErrors since the data is formatted differently than the default QDhourly. I tried reformatting the data myself but that didn't work either. Is there a better way to calculate L-shell past Jan. 2022?

@drsteve
Copy link
Member

drsteve commented Mar 13, 2024

The "OMNI2" is taken directly from NASA's OMNI2 data set (the 2 distinguishes it from the ~70s era OMNI product). The "OMNI" data set used by SpacePy is a derived product that's more widely known - at least these days - as Qin-Denton data. Hence the QD in the QDhourly name.

As noted previously ViRBO were generating and serving that. After ViRBO ceased to generate the data, LANL picked up the processing and intermittently passed updates back to ViRBO to serve. At some point the processing and provision was passed to the Van Allen Probes ECT suite since they had support for a website and data processing, as well as a need for the data product. So the current update code pulls the older data from ViRBO and then pulls the hourly-resolution (text) files from the ECT website to build a combined data set.

The issue is that Van Allen Probes ended and post-mission support has effective ended too. So there's currently no current public-facing generation of the Qin-Denton data products past January 2022. They are used by the NASA MMS mission in generation of the MEC data product (at higher resolution, but we generate 1, 5 and 60 minute files), but the underlying files aren't served. We're currently looking for a more permanent home for the ongoing data processing to generate that data product as well as to serve it. SpacePy doesn't have funding to support ongoing processing and data hosting.

As a temporary fix I can grab the post-2022 files and find a spot to make them available. It'll probably end up being a dump on Zenodo or something similar. I'll update here once I have the best path for that figured out.

@candeynasa
Copy link

The QD data can be archived at SPDF; adding to CDAWeb would also need a version in CDF form with ISTP metadata. If only the OMNI values are needed, the ASCII files for OMNI are in the high and low res directories under https://spdf.gsfc.nasa.gov/pub/data/omni/ . I mentioned the OMNI CDFs since they are generally easier to load (such as with pycdf https://spacepy.github.io/pycdf.html ) or even easier direct from CDAWeb using its API via cdasws https://cdaweb.gsfc.nasa.gov/WebServices/REST/ .

@julia-claxton
Copy link
Author

julia-claxton commented Apr 2, 2024 •
edited
Loading

Hi, I'm sorry if this is a silly question. I am unsure how to download the OMNI data from the websites provided in such a way that spacepy can use them for L-shell calculations. The files don't appear to be formatted in the same way as the QDHourly data that spacepy uses, which causes spacepy to throw KeyErrors when it looks at the provided data that is formatted differently than the Qin-Denton data. I find myself still unable to use irbempy.get_Lm() for any time after Jan 2022 to get L-shell from position data.

What would be the best way going about getting L-shell from position & time after Jan 2022? If using the differently-formatted OMNI data, should I reformat it myself or is there a setting in spacepy I can use?

@jtniehof
Copy link
Member

jtniehof commented Apr 3, 2024

@julia-claxton, right now the best option is unfortunately to wait until Steve can get things squared away. The original OMNI CDF files at CDAWeb/SPDF do not have some of the derived quantities which are in the QD files, which is why we use the QD. The OMNI2 data, in addition to having a different set of names than OMNI/QD, also don't have those derived quantities, so effectively they can't be used as inputs to the IRBEM routines. See #617.

@candeynasa , if you're offering to host the QD files at SPDF, and hopefully even figure a means of routine production, we should talk. We need a low-friction way to keep these things up to date and it's been hampered by being on a volunteer basis.

@jtniehof
Copy link
Member

@drsteve, @candeynasa, and others at SPDF have been working on hosting. In the meantime, Steve generated up-to-date files and I've processed them into the final parsed SpacePy database. That can be downloaded here. Place that .h5 file in the data directory inside the Spacepy DOT_FLN directory.

@julia-claxton
Copy link
Author

That worked perfectly, thank you so much @jtniehof!

@jtniehof jtniehof mentioned this issue Oct 8, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jtniehof @drsteve @julia-claxton @candeynasa