Question

can we use offset in amazon s3 select query.

  • 28 July 2023
  • 5 replies
  • 159 views

Hi Team,
We are using groovy_scripting origin to read files from s3 bucket. But problem is that we are not able to apply offset in sql query used in code. Can u help us to provide some way to apply offset in sql query. I am attaching code snippet that is working but we are not able to apply offset in it.

 


5 replies

Userlevel 5
Badge +1

@himanshu1234567 

I got this below code snippet from google with sql query on s3 object and it works fine for it.

May i know if the code is working fine in your local setup .

#  start client with s3
s3 = boto3.client('s3', region_name='us-east-1')

# define file location and name
bucket_name = 'MY-BUCKET-NAME'
filename = 'test.csv'

# create SQL expression to query by date using column index
sql_exp = "SELECT * FROM s3object s LIMIT 1"

# should we use header names to filter
use_header = False

# return CSV of unpacked data
file_str = query_csv_s3(s3, bucket_name, filename, sql_exp, use_header)

# read CSV to dataframe
df = pd.read_csv(StringIO(file_str))

# print column names
print(list(df.columns))

 

@Bikram where u are using offset here, my problem is that sql query is not supporting offset.Can we use offset in your code?

@Bikram  we need groovy/java code?

Userlevel 5
Badge +1

@himanshu1234567 

groovy only.

@Bikram above code u pasted is groovy?

Reply