Confluence Retirement

Due to the feedback from stakeholders and our commitment to not adversely impact USGS science activities that Confluence supports, we are extending the migration deadline to January 2023.

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is targeted for retirement. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.
Community for Data Integration
Space shortcuts

Home

Page tree

Versions Compared

Old Version 3

changes.mady.by.user Viger, Roland

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (cprice@usgs.gov)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: wordsmithing the preamble

Pandas is an incredibly convenient module that can be used in Pandas. Its Python module for working with tabular data when ArcGIS table tools and workflows are missing functionality or are simply too slow. Panda's main data structure, the DataFrame, cannot be directly ingested back into a GDB table. First you have to convert it to a numpy arrayEsri's tool to do this, NumPyArrayToTable(), only reads numpy arrays. More specifically, this needs to be the tool requires a structured numpy array, which means that each column needs to have a "dtype" definition that specifies the name and data type (like "int16"). 

There are several problems, the first of which is that the conversion from the pandas.DataFrame to a numpy.array tends to strip off the specification of the data type in each column, or at least it does some of the things you want, but not all. When making a pandas-->numpy conversion, each column is cast from a specific pandas data type to a corresponding numpy data type. The crux of the problem is that there are not equivalent numpy types for all pandas data types (most, but definitely not all). In these cases, the conversion "upcasts" to a more generic data type. This frequently results in an output data type of "object", which is so generic that the arcpy.da.NumPyArrayToTable() function barfs. This conversion to "object" happens for string fields, for example. If the pandas.DataFrame columns have different data types, then the conversion usually just treats all columns as "object".

...