#PYTHON #DATABASE #MYSQL #VERTICA #PARQUST #CSV
def to_parquet(engine,
sql_query: str,
file_name: os.PathLike,
compression=PARQUET_COMPRESSION_SNAPPY,
func_print: Callable = print) -> int
SQL Query Statemet to Parquet format file.
Arguments:
engine
type - Connection Database And SQLAlchemy.Enginesql_query
str - SQL Query Statement (SELECT Only)file_name
os.PathLike - save with filename and extention file (Example: ./myparquet.parquet
)compression
_type, optional_ - description. Compression file type to PARQUET_COMPRESSION_SNAPPY.func_print
Callable, optional - Callback Print Massage function . Defaults to print.Raises:
ex
- Errror HandlerReturns:
int
- Total count record data.def read_parquet(filename: os.PathLike) -> pd.DataFrame
Read Parquet file into pandas DataFrame
Arguments:
filename
os.PathLike - file name os.PathLike
Returns:
pd.DataFrame
- pandas DataFramedef head_parquet(filename: os.PathLike, batch_size: int = 10) -> pd.DataFrame
Read Head record in Parquet file
Arguments:
filename
os.PathLike - filenamenrows
int, optional - number rows. Defaults to 10.Returns:
pd.DataFrame
- pandas DataFramedef batch_parquet(filename: os.PathLike,
batch_size: int = 10000) -> tp.Iterator[pd.DataFrame]
Read Parquet file into iteration pandas dataframe object
Arguments:
filename
os.PathLike - filenamebatch_size
int, optional - batch_size or chunksize row number. Defaults to 10000.Yields:
Iterator[pd.DataFrame]
- Return Iterator[pd.DataFrame]