聚合国内IT技术精华文章,分享IT技术精华,帮助IT从业人士成长

  • 2209591 views阅读

    A BigQuery error about the partition

    We were using client.query() (from Python API of BigQuery) to insert selected data into a table with a specific partition. But the script reported errors like: google.api_core.exceptions.BadReq...

    分类:技术文章 时间:2023-01-06 15:15 我要评论(0个)

  • 2768882 views阅读

    Using Python to run BigQuery job with project id

    Here is the code for me to query a table of BigQuery: from google.cloud import bigquery from google.cloud.bigquery_storage import BigQueryReadClient client = bigquery.Client() storage_client =...

    分类:技术文章 时间:2022-11-18 10:30 我要评论(0个)

  • 2501038 views阅读

    A strange error in BigQuery

    Two days ago we met a weird error when running a select through BigQuery Python API: Error : google.api_core.exceptions.BadRequest: 400 Bad int64 value: BA1D I checked the select SQL but it ...

    分类:技术文章 时间:2022-09-15 18:59 我要评论(0个)

  • 2826630 views阅读

    The correct way to insert data from another table in BigQuery

    Incorrect code: WITH source1 as ( SELECT blah FROM blah ), source2 as ( SELECT moreblah FROM source1 ) INSERT INTO newtable FROM source2; Correct solution: INSERT INTO newtable WITH s...

    分类:技术文章 时间:2022-08-12 08:44 我要评论(0个)

  • 2392766 views阅读

    pandas.datetime64 with Timezone

    p>I barely pay attention to the pandas.datetime64 type. But yesterday a problem stroke me. It was a parquet file with a column “start_date”: >>> df["start_date"...

    分类:技术文章 时间:2022-03-25 13:46 我要评论(0个)

  • 2081086 views阅读

    Get DDL of a table in BigQuery

    How could I conveniently get the creating-SQL of a table in BigQuery? We could use INFORMATION_SCHEMA: SELECT table_name, ddl FROM `data-to-insights.taxi.INFORMATION_SCHEMA.TABLES` WHERE ta...

    分类:技术文章 时间:2021-10-22 10:31 我要评论(0个)

  • 1295433 views阅读

    Recover truncated table in BigQuery

    If you accidentally truncate a table in BigQuery, you can try this article to recover the data. Furthermore, I found out that the "bq cp project:dataset.table@-36000 project:dataset.table” m...

    分类:技术文章 时间:2021-06-03 15:08 我要评论(0个)

  • 1779224 views阅读

    Migrate Spark job to BigQuery

    I have just finished a work about migrating Spark job to BigQuery, or more precisely: migrate Python code to SQL. It’s a tedious work but improve the performance significantly: from 4 hours ...

    分类:技术文章 时间:2021-05-07 11:47 我要评论(0个)

  • 1977665 views阅读

    Change the schema of BigQuery tables

    We can easily add new column for a table in BigQuery: ALTER TABLE mydataset.mytable ADD COLUMN new_col STRING But when you want to delete or rename an existed column, there is no SQL to imp...

    分类:技术文章 时间:2021-03-11 12:58 我要评论(0个)

  • 1584556 views阅读

    A few notes for Pandas and BigQuery

    Get the memory size of a DataFrame of Pandas df.memory_usage(deep=True).sum() 2. Upload a large DataFrame of Pandas to BigQuery table If your DataFrame is too big, the uploading operation...

    分类:技术文章 时间:2021-01-22 10:29 我要评论(0个)

  • 1663034 views阅读

    Import date column in Pandas to BigQuery

    Imaging we have a small CSV file: name,enroll_time robin,2021-01-15 09:50:33 tony,2021-01-14 01:50:33 jaime,2021-01-13 00:50:33 tyrion,2021-2-15 13:22:17 bran,2022-3-16 14:00:01 Let’s ...

    分类:技术文章 时间:2021-01-15 12:26 我要评论(0个)

  • 2275534 views阅读

    Compare two tables in BigQuery

    As this answer, the best solution for comparing two tables in BigQuery is: ( SELECT * FROM table1 EXCEPT DISTINCT SELECT * from table2 ) UNION ALL ( SELECT * FROM table2 EXCEPT DISTINCT ...

    分类:技术文章 时间:2020-09-17 20:32 我要评论(0个)