Table of Contents

• dbhelper
• parquet
  • to_parquet
  • read_parquet
  • head_parquet
  • batch_parquet

parquet

to_parquet

def to_parquet(engine,
               sql_query: str,
               file_name: os.PathLike,
               compression=PARQUET_COMPRESSION_SNAPPY,
               func_print: Callable = print) -> int

SQL Query Statemet to Parquet format file.

Arguments:

• engine type - Connection Database And SQLAlchemy.Engine
• sql_query str - SQL Query Statement (SELECT Only)
• file_name os.PathLike - save with filename and extention file (Example: ./myparquet.parquet)
• compression _type, optional_ - description. Compression file type to PARQUET_COMPRESSION_SNAPPY.
• func_print Callable, optional - Callback Print Massage function . Defaults to print.

Raises:

• ex - Errror Handler

Returns:

• int - Total count record data.

read_parquet

def read_parquet(filename: os.PathLike) -> pd.DataFrame

Read Parquet file into pandas DataFrame

Arguments:

• filename os.PathLike - file name os.PathLike

Returns:

• pd.DataFrame - pandas DataFrame

head_parquet

def head_parquet(filename: os.PathLike, batch_size: int = 10) -> pd.DataFrame

Read Head record in Parquet file

Arguments:

• filename os.PathLike - filename
• nrows int, optional - number rows. Defaults to 10.

Returns:

• pd.DataFrame - pandas DataFrame

batch_parquet

def batch_parquet(filename: os.PathLike,
                  batch_size: int = 10000) -> tp.Iterator[pd.DataFrame]

Read Parquet file into iteration pandas dataframe object

Arguments:

• filename os.PathLike - filename
• batch_size int, optional - batch_size or chunksize row number. Defaults to 10000.

Yields:

• Iterator[pd.DataFrame] - Return Iterator[pd.DataFrame]