Alibabacloud Odps Maxframe Coding

Common Workflow Complete Guide

Detailed guide for the complete MaxFrame development workflow with comprehensive examples.

Session Setup Patterns

Pattern 1: Auto-detect (DataWorks/MaxCompute Notebook)

import os
import maxframe.dataframe as md
from maxframe.session import new_session
from odps import ODPS

# Auto-detect from environment (preferred in DataWorks/MaxCompute Notebook)
session = new_session()

Pattern 2: Explicit ODPS Connection

import os
import dotenv
import maxframe.dataframe as md
from maxframe.session import new_session
from odps import ODPS

dotenv.load_dotenv()

o = ODPS(
    access_id=os.getenv("ODPS_ACCESS_ID"),
    secret_access_key=os.getenv("ODPS_ACCESS_KEY"),
    project=os.getenv("ODPS_PROJECT"),
    endpoint=os.getenv("ODPS_ENDPOINT"),
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)
session = new_session(o)

Pattern 3: Production-ready Session

import logging
import maxframe.dataframe as md
from maxframe.session import new_session

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

session = new_session()
try:
    logger.info(f"Session created. Logview: {session.get_logview_address()}")
    # Your operations
    ...
finally:
    session.destroy()
    logger.info("Session destroyed")

Reading Data Patterns

Pattern 1: Basic Table Read

# Read from MaxCompute table
df = md.read_odps_table("table_name")

# Read with index column
df = md.read_odps_table("table_name", index_col="id")

# With column selection
df = md.read_odps_table("table_name", columns=['id', 'value', 'timestamp'])

# With partition filter
df = md.read_odps_table("table_name", partition='ds=2024-01-01')

Pattern 2: SQL Query Read

# Read from SQL query with filters
df = md.read_odps_query(
    "SELECT * FROM table WHERE date >= '2024-01-01' AND status = 'active'"
)

# Complex SQL with joins
df = md.read_odps_query(
    "SELECT a.*, b.value FROM table_a a JOIN table_b b ON a.id = b.id"
)

Pattern 3: Sample Data Construction

When user doesn't provide input table name, construct pandas DataFrame:

import pandas as pd
import numpy as np

# Time series analysis example
example_pd_df = pd.DataFrame({
    'timestamp': pd.date_range('2026-01-01', periods=1000, freq='H'),
    'metric_name': np.random.choice(['cpu', 'memory', 'disk'], 1000),
    'value': np.random.randn(1000) * 10 + 50,
    'host_id': np.random.choice(['host1', 'host2', 'host3'], 1000)
})

# Load into MaxFrame
df = md.read_pandas(example_pd_df)

Key guidelines for sample data:

• Match data types and structure to job requirements
• Use realistic value ranges for the domain
• Include 100-1000 rows to demonstrate logic
• Use descriptive column names matching operations

Operator Selection Workflow

Step 1: Identify Required Operations

Break down the task into specific operations needed:

• Filtering
• Grouping
• Aggregation
• Transformation
• Merging
• Sorting

Step 2: Find MaxFrame Operators

Use operator-selector agent or script:

# Search for operators by task description
python scripts/lookup_operator.py search "time series resampling"

# Check if a specific operator exists
python scripts/lookup_operator.py info apply_chunk

# Get detailed operator information
python scripts/lookup_operator.py info groupby

Step 3: Present Options to User

For your data aggregation task, I've identified these options:

1. `groupby().agg()` - Standard pandas-compatible approach
   - Pros: Familiar API, good for standard aggregations
   - Cons: May be slow for large datasets with custom logic

2. `mf.apply_chunk()` - For custom aggregation with large datasets
   - Pros: Efficient batch processing, custom logic support
   - Cons: More complex, requires batch size tuning

Which approach do you prefer, or would you like me to explore other options?

Step 4: Get User Confirmation

MANDATORY: Do not proceed without user confirmation.

Processing Patterns

Pattern 1: Standard pandas Operations

# Filter
filtered = df[df['column'] > 10]

# GroupBy and aggregate
result = df.groupby('category').agg({'value': 'sum'})

# Add columns
df['new_col'] = df['col1'] + df['col2']

# Sort
df_sorted = df.sort_values('column')

# Merge
df_merged = df1.merge(df2, on='key')

# Multiple aggregations
result = df.groupby('category').agg({
    'value': ['sum', 'mean', 'count'],
    'price': 'max'
})

Pattern 2: Batch Processing (Large Datasets)

def process_batch(chunk):
    # Custom processing logic
    return chunk * 2

result = df.mf.apply_chunk(
    process_batch,
    batch_rows=1024,  # Tune batch size for performance
    output_type='dataframe'
)

Pattern 3: UDF with Resource Allocation

from maxframe.udf import with_running_options

@with_running_options(engine="dpe", cpu=2, memory=4)
def process_batch(batch):
    # CRITICAL: memory=4 means 4 GB, NOT 4 MB
    return batch * 2

result = df.mf.apply_chunk(process_batch)

Writing Data Patterns

Pattern 1: Write to MaxCompute Table

# Write to MaxCompute table
md.to_odps_table(df, "output_table", overwrite=True).execute()

Pattern 2: Write to DLF External Table

from maxframe import options

# Enable DLF support
options.sql.settings = {
    "odps.maxframe.resolve_dlf_tables": "true"
}

md.to_odps_table(df, "dlf_table").execute()

Pattern 3: Multiple Output Tables

try:
    md.to_odps_table(df1, "output_table1").execute()
    md.to_odps_table(df2, "output_table2").execute()
finally:
    session.destroy()

Execution and Cleanup Patterns

Pattern 1: Basic Execution

# Execute operations (required for lazy execution)
result.execute()

# Destroy session when done
session.destroy()

Pattern 2: Safe Cleanup (Production)

try:
    # Execute operations
    result.execute()
finally:
    # Destroy session (always runs, even on error)
    session.destroy()

Pattern 3: Comprehensive Cleanup

import logging

logger = logging.getLogger(__name__)

try:
    result.execute()
    logger.info("Execution successful")
except Exception as e:
    logger.error(f"Execution failed: {e}")
    raise
finally:
    try:
        session.destroy()
        logger.info("Session destroyed")
    except Exception as cleanup_error:
        logger.warning(f"Cleanup error: {cleanup_error}")

Verification Pattern

Use py_compile to test generated job script:

python -m py_compile your_script.py

Complete Example Pipeline

import os
import logging
import dotenv
import maxframe.dataframe as md
from maxframe.session import new_session

dotenv.load_dotenv()
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Setup session
session = new_session()
try:
    logger.info(f"Session created. Logview: {session.get_logview_address()}")

    # Read data
    df = md.read_odps_table("source_table", columns=['id', 'value', 'category'])

    # Process (after confirming operators with user)
    filtered = df[df['value'] > 100]
    result = filtered.groupby('category').agg({'value': 'sum'})

    # Write output
    md.to_odps_table(result, "output_table", overwrite=True).execute()

    logger.info("Job completed successfully")
    logger.info(f"Final Logview: {session.get_logview_address()}")

finally:
    session.destroy()
    logger.info("Session destroyed")

MaxFrame Installation Guide

This guide provides step-by-step instructions for installing and configuring MaxFrame for distributed data processing on MaxCompute.

Table of Contents

• Prerequisites
• Dependencies
• Environment Configuration
• Required Environment Variables
• Setting Environment Variables
• Find Your MaxCompute Endpoint
• Installation Verification
• Session Setup
• Manual Session Creation
• Auto-Detect from Environment
• Troubleshooting
• Common Issues
• Getting Help
• Next Steps
• Cleanup

Prerequisites

• Python 3.7 or higher
• MaxCompute (ODPS) account with valid credentials
• Access to a MaxCompute project

Dependencies

Install the required Python packages:

pip install maxframe -U

The required packages are:

• maxframe - MaxFrame SDK for distributed data processing
• pyodps - ODPS Python SDK for MaxCompute access
• pandas - Data manipulation library (for pandas-compatible APIs)

Environment Configuration

Required Environment Variables

Configure the following environment variables to authenticate with MaxCompute:

Setting Environment Variables

Option 1: Set in Shell

export ODPS_ACCESS_ID="your_access_id"
export ODPS_ACCESS_KEY="your_access_key"
export ODPS_PROJECT="your_project_name"
export ODPS_ENDPOINT="your_endpoint"

Option 2: Use .env File

Create a .env file in your project directory:

ODPS_ACCESS_ID=your_access_id
ODPS_ACCESS_KEY=your_access_key
ODPS_PROJECT=your_project_name
ODPS_ENDPOINT=your_endpoint

Then load the environment variables in Python:

from dotenv import load_dotenv

load_dotenv()

Find Your MaxCompute Endpoint

MaxCompute endpoints vary by region, check the MaxCompute documentation for the correct endpoint for your region.

Installation Verification

Verify your installation by running the following Python script:

import os
from dotenv import load_dotenv
from odps import ODPS
from maxframe.session import new_session

# Load environment variables
load_dotenv()

# Create ODPS connection
o = ODPS(
    access_id=os.getenv("ODPS_ACCESS_ID"),
    secret_access_key=os.getenv("ODPS_ACCESS_KEY"),
    project=os.getenv("ODPS_PROJECT"),
    endpoint=os.getenv("ODPS_ENDPOINT"),
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)

# Create MaxFrame session
session = new_session(o)

print("MaxFrame installation verified successfully!")
print(f"Connected to project: {o.project}")

# Destroy session when done
session.destroy()

Session Setup

Manual Session Creation

Create a session with explicit credentials:

import os
import maxframe.dataframe as md
from maxframe.session import new_session
from odps import ODPS

# Create ODPS connection
o = ODPS(
    access_id=os.getenv("ODPS_ACCESS_ID"),
    secret_access_key=os.getenv("ODPS_ACCESS_KEY"),
    project=os.getenv("ODPS_PROJECT"),
    endpoint=os.getenv("ODPS_ENDPOINT"),
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)

# Create MaxFrame session
session = new_session(o)

Auto-Detect from Environment

In environments like DataWorks or MaxCompute Notebook, ODPS credentials are automatically available:

from maxframe.session import new_session

# Auto-detects ODPS from environment
session = new_session()

Troubleshooting

Common Issues

Issue: Connection Authentication Failed

Symptoms: Error message indicating invalid credentials or authentication failure.

Solutions:

• Verify all environment variables are set correctly
• Check that your access key has not expired
• Ensure you have the correct endpoint for your region
• Verify your project name is accurate

# Test environment variables
echo $ODPS_ACCESS_ID
echo $ODPS_PROJECT
echo $ODPS_ENDPOINT

Issue: Package Installation Fails

Symptoms: pip install fails with dependency conflicts or permission errors.

Solutions:

• Use a virtual environment to isolate dependencies:

python -m venv maxframe_env
source maxframe_env/bin/activate  # On Windows: maxframe_env\Scripts\activate
pip install maxframe pyodps pandas --prefer-binary

• Upgrade pip before installing:

pip install --upgrade pip
pip install maxframe pyodps pandas --prefer-binary

Issue: Session Creation Fails

Symptoms: new_session() raises an exception.

Solutions:

• Verify network connectivity to the MaxCompute endpoint
• Check firewall rules allow outbound connections
• Ensure your MaxCompute account has the necessary permissions
• Try the auto-detect method if available in your environment
• Use VPC endpoint if you are in vpc networking

Issue: Lazy Execution Not Working

Symptoms: Operations appear to do nothing until .execute() is called.

Note: This is expected behavior. MaxFrame uses lazy execution. Always call .execute() to trigger computation:

# This does not execute immediately
result = df.groupby('category').sum()

# Execute the computation
result.execute()

Getting Help

If you encounter issues not covered here:

1. Check the MaxFrame Documentation 2. Review the MaxFrame Client Repository 3. Consult the sample code in assets/examples/ for working examples 4. Contact your MaxCompute administrator for account-specific issues

Next Steps

After successful installation:

1. Review the MaxFrame Context Guide for comprehensive feature documentation 2. Explore the sample code for working examples 3. Start building your first MaxFrame program using the Common Workflow

Cleanup

Destroy your session when done to free resources:

session.destroy()

MaxFrame Local Debug Mode Guide

This guide provides comprehensive instructions for using MaxFrame's local debug mode, which enables offline UDF development with full IDE debugging support.

Overview

MaxFrame Local Debug Mode is designed for data development engineers to debug UDF (User-Defined Functions) locally without connecting to remote MaxCompute services. It provides a seamless development experience with IDE breakpoint support for functions like apply() and apply_chunk().

Core Value

Key Benefits

1. Zero-Configuration Startup: Simply use debug=True or debug="local" - no additional tools or services required 2. Completely Offline: No dependency on network or remote cluster resources 3. Native IDE Support: Breakpoints, variable inspection, step-by-step execution - all debugging capabilities preserved 4. Flexible Data Sources: Support for in-memory data, local files, or MaxCompute tables 5. Seamless Production Switch: Remove debug=True parameter and code runs directly in production

When to Use Local Debug Mode

Use local debug mode when:

• Developing UDF functions (apply, apply_chunk)
• Need IDE breakpoints and step-by-step debugging
• Want to debug offline without network access
• Working on complex logic that requires iterative testing
• Need to verify data transformation logic quickly

Use remote debug mode instead when:

• Testing with production-scale data on MaxCompute
• Need to verify execution on actual cluster
• Investigating runtime issues that require logview URLs
• Debugging distributed execution problems

Quick Start

Prerequisites

pip install --upgrade maxframe  # Requires MaxFrame SDK 2.5.0 or later

Basic Example

from odps import ODPS
from maxframe import new_session
import maxframe.dataframe as md
import pandas as pd

# Initialize ODPS object
# Note: In local debug mode, ODPS object is only used for schema validation
# Actual credentials are not used for execution
o = ODPS(
    access_id=os.getenv('ODPS_ACCESS_ID', 'dummy_access_id'),
    secret_access_key=os.getenv('ODPS_ACCESS_KEY', 'dummy_secret_key'),
    project=os.getenv('ODPS_PROJECT', 'dummy_project'),
    endpoint=os.getenv('ODPS_ENDPOINT', 'dummy_endpoint'),
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)

# Enable local debug mode
session = new_session(o, debug=True)

# Prepare sample data
df = md.DataFrame(pd.DataFrame({
    "sales": [5000, 8000, 12000, 3000],
    "region": ["A", "B", "C", "D"]
}))

def calculate_commission(row):
    sales = row['sales']
    if sales > 10000:  # Set breakpoint here
        rate = 0.15
        print(rate)
    elif sales > 5000:  # Set breakpoint here
        rate = 0.10
        print(rate)
    else:
        rate = 0.05
    return sales * rate

# Execute and get results
result = df.apply(calculate_commission, axis=1).execute().fetch()
print(result)

Key Features

1. Zero-Configuration Startup

Simply add debug=True or debug="local" when creating a session:

# Local debug mode
session = new_session(o, debug=True)
# or
session = new_session(o, debug="local")

# Production mode (just remove debug parameter)
session = new_session(o)

2. IDE-Friendly Debugging

• Supported IDEs: PyCharm, VSCode, and other mainstream IDEs, as well as DataWorks Notebook
• Breakpoints: Set breakpoints anywhere in your UDF functions
• Step-by-Step Execution: Use F5/F6/F7/F8 to navigate through code
• Variable Inspection: View and modify variables during debugging
• Debugging Experience: Identical to local Python development

3. Multiple Data Sources

Example with different data sources:

# 1. In-memory data (fastest for testing)
import pandas as pd
df = md.DataFrame(pd.DataFrame({
    "col1": [1, 2, 3],
    "col2": ["a", "b", "c"]
}))

# 2. MaxCompute table (real data)
df = md.read_odps_table("your_table_name")

# 3. Local file (offline development)
local_df = pd.read_csv("local_data.csv")
df = md.read_pandas(local_df)

4. Code Compatibility

Debugging code is identical to production code. Simply remove the debug parameter when deploying:

# Development environment
session = new_session(o, debug=True)
# ... your code ...

# Production environment
session = new_session(o)
# ... same code ...

Application Scenarios

Important Considerations

1. Performance Differences

Local debug mode is designed for development and verification. Performance characteristics differ from production environment:

• Execution happens locally, not distributed
• Performance is not representative of production cluster performance
• Best suited for small-scale sample data

2. Data Volume Limitations

For optimal debugging experience:

• Use small-scale sample data (recommended: 100-1000 rows)
• Large datasets may slow down local execution
• Focus on logic correctness rather than performance

3. Dependency Consistency

Ensure local Python environment matches production:

• Same Python version
• Same package versions (maxframe, pandas, numpy, etc.)
• Use pip freeze > requirements.txt to capture dependencies

4. Sensitive Data Handling

When debugging with MaxCompute tables:

• Be aware of data permissions and access controls
• Consider data masking for sensitive information
• Use sample/partitioned data to limit exposure
• Never commit sensitive credentials to version control

Common Debugging Patterns

Pattern 1: Breakpoint in Apply Function

def process_row(row):
    # Set breakpoint on this line
    value = row['column_name']

    if value > threshold:
        # Set breakpoint here to inspect condition
        result = transform(value)
    else:
        result = default_value

    return result

df = md.DataFrame(sample_data)
result = df.apply(process_row, axis=1).execute().fetch()

Pattern 2: Debugging Apply_Chunk for Batch Processing

def process_batch(chunk):
    # Set breakpoint here to inspect entire chunk
    print(f"Processing batch with {len(chunk)} rows")

    # Debug data types
    print(f"Chunk dtypes:\n{chunk.dtypes}")

    # Debug transformations
    chunk['new_col'] = chunk['col1'] * 2

    # Set breakpoint here to verify results
    return chunk

result = df.mf.apply_chunk(
    process_batch,
    batch_rows=100,
    output_type='dataframe'
).execute().fetch()

Pattern 3: Debugging with Print Statements

def debug_function(row):
    print(f"Input row: {row.to_dict()}")

    # Step 1
    intermediate = row['col1'] + row['col2']
    print(f"After step 1: {intermediate}")

    # Step 2
    result = intermediate * 2
    print(f"Final result: {result}")

    return result

# Execute with debug output
result = df.apply(debug_function, axis=1).execute().fetch()

Transitioning to Production

Steps to Deploy

1. Test Locally: Develop and debug with local debug mode 2. Verify Logic: Ensure all transformations work correctly 3. Remove Debug Parameter: Change new_session(o, debug=True) to new_session(o) 4. Test on Cluster: Run on MaxCompute with small dataset 5. Production Deploy: Deploy to production environment

Code Checklist

Before deploying to production:

• [ ] Remove debug=True parameter from session creation
• [ ] Verify all data source paths are correct for production
• [ ] Test with production-scale data on MaxCompute
• [ ] Remove or reduce print statements used for debugging
• [ ] Add proper error handling and logging
• [ ] Verify resource quotas and permissions

Troubleshooting

Issue: IDE Breakpoints Not Triggering

Possible Causes:

• Session created without debug=True
• Using incompatible IDE or debugger
• Code not actually executing through apply/apply_chunk

Solutions:

• Verify debug=True in new_session()
• Ensure you're using a supported IDE (PyCharm, VSCode)
• Check that .execute() is called to trigger execution

Issue: Local Execution Too Slow

Possible Causes:

• Dataset too large for local debugging
• Complex operations not optimized for local execution

Solutions:

• Reduce sample data size (use df.head(100) or sample)
• Simplify operations for debugging purposes
• Focus on specific problematic code sections

Issue: Results Differ from Production

Possible Causes:

• Data differences between sample and production data
• Environmental differences (Python version, package versions)
• Distributed vs. local execution semantics

Solutions:

• Verify data consistency between environments
• Check Python and package versions match
• Test on MaxCompute with debug=False to verify

Summary

Local debug mode provides a powerful development experience for MaxFrame UDF development:

• Zero-configuration startup with debug=True
• Full IDE debugging support with breakpoints
• Flexible data source options
• Seamless transition to production
• Perfect for iterative UDF development

Use local debug mode during development for rapid iteration, then switch to remote debug mode for cluster-based testing and validation.

Resources

• MaxFrame Context Guide: ./maxframe-context.md - Comprehensive MaxFrame features and workflows
• Interactive Coding Guide: ./remote-debug-guide.md - Remote debug mode with logview support
• Key Modules Reference: ./key-modules.md - DataFrame, Tensor, and ML operations

Comparison with other tools

• Comparison with PyODPS DataFrame
• Object abstraction
• Functions
• Execution

Comparison with PyODPS DataFrame

PyODPS DataFrame is a DataFrame-like package provided by MaxCompute as a part of PyODPS package. It provides capability for Python data analyzers to query MaxCompute data with a set of operators similar to pandas. Despite the similarity in operators, the usage between two sets of APIs are quite different. It might not be easy for a developer to dive deep into PyODPS DataFrame with knowledge about pandas only.

Though PyODPS DataFrame is still part of PyODPS, it is recommended to create new applications with MaxFrame to enjoy its compatibility with pandas.

Object abstraction

PyODPS DataFrame does not have indexes. This means that a majority of pandas APIs with indexes cannot be used or not fully supported.

For instance, arithmetic operations in pandas relies on index alignment. That is, two DataFrames are aligned first, and then arithmetic operation is performed.

>>> series1 = pd.Series([2, 1, 3], index=[1, 2, 4])
>>> series2 = pd.Series([1, 5, 6], index=[1, 3, 4])
>>> series1 + series2
1    3.0
2    NaN
3    NaN
4    9.0
dtype: float64

However, when indexes are absent, this kind of operation is not supported.

To support this kind of operation, in MaxFrame, it is required to add an index column to DataFrame or Series. If the index is absent, a default RangeIndex is added. Therefore the statement above can be supported.

Another huge difference between PyODPS DataFrame and MaxFrame is that in PyODPS DataFrame, representation of data objects and operators are mixed, and this may confuse newcomers. For instance,

df = o.get_table('table_name').to_df()  # df is a DataFrame instance
df2 = df["col1", "col2"]  # df2 is a CollectionExpr instance

In the second line, df2 is an instance of CollectionExpr which means it is an expression and different from a DataFrame instance. However, all DataFrame functions can be applied directly onto df2 and there is nothing different from DataFrame instance.

In MaxFrame, however, data objects and operators are defined separately. Data objects users interact with are all instances of a few data classes, namely DataFrame, Series or Index. For the example above, now all instances are DataFrame now.

df = md.read_odps_table('table_name')  # df is a DataFrame instance
df2 = df[["col1", "col2"]]  # df2 is also a DataFrame instance

Functions

Functions in PyODPS DataFrame are not fully compatible with pandas. Therefore to write code with PyODPS DataFrame, users need to read the documents first before start coding. However, the target of MaxFrame is to create a pandas-compatible API. Hence there are API differences between PyODPS DataFrame and MaxFrame. These differences are listed below. Methods starts with mf. mean that these non-pandas methods are added in MaxFrame to facilitate migrating from PyODPS DataFrame to MaxFrame. Note that you need to read API documents of these functions before rewriting your code.

Execution

PyODPS DataFrame and MaxFrame both use lazy execution to leverage efficiency of code optimization. However, the way to invoke these jobs is changed.

<a id="getting-started-index"></a>

Getting Started

• Access and installation
• Enable MaxFrame for your MaxCompute project
• Install MaxFrame client locally
• Access MaxFrame with DataWorks
• Access MaxFrame with MaxCompute Notebook
• Overview
• Getting started tutorials
• 10 minutes to MaxFrame
• Comparison with other tools
• Comparison with PyODPS DataFrame

Access and installation

Enable MaxFrame for your MaxCompute project

You need to setup a MaxCompute project Before using MaxFrame. Please take a look at here for more information.

NOTE

Currently MaxFrame is under trial. If you need to enable MaxFrame for your MaxCompute project, please fill the form to apply for trial here.

Install MaxFrame client locally

After created your own MaxCompute project and enabled MaxFrame, you may install MaxFrame client with pip command:

pip install maxframe

Then you can create a MaxCompute table, perform some transformation with MaxFrame and then store the result into another MaxCompute table.

import maxframe.dataframe as md
from odps import ODPS
from maxframe import new_session

# create MaxCompute entrance object and test table
o = ODPS(
    access_id=os.getenv('ODPS_ACCESS_ID'),
    secret_access_key=os.getenv('ODPS_ACCESS_KEY'),
    project='your-default-project',
    endpoint='your-end-point',
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)
table = o.create_table("test_source_table", "a string, b bigint")
with table.open_writer() as writer:
    writer.write([
        ["value1", 0],
        ["value2", 1],
    ])

# create maxframe session
session = new_session(o)

# perform data transformation
df = md.read_odps_table("test_source_table")
df["a"] = "prefix_" + df["a"]
md.to_odps_table(df, "test_prefix_source_table").execute()

# destroy maxframe session
session.destroy()

Access MaxFrame with DataWorks

DataWorks provides task scheduling capability for MaxCompute projects. You can schedule and run MaxFrame job with DataWorks.

To run MaxFrame job with DataWorks, you need to create a PyODPS 3 node and write your code inside it. PyODPS nodes are executed with embedded MaxCompute accounts and project information, thus you may create your MaxFrame session directly.

import maxframe.dataframe as md
from maxframe import new_session

# create maxframe session
session = new_session(o)

# perform data transformation
df = md.read_odps_table("test_source_table")
df["a"] = "prefix_" + df["a"]
md.to_odps_table(df, "test_prefix_source_table").execute()

# destroy maxframe session
session.destroy()

Access MaxFrame with MaxCompute Notebook

MaxCompute Notebook also provides MaxFrame package. It also provides MaxCompute account in environment variables in the notebook, thus account information is not needed.

import maxframe.dataframe as md
from maxframe import new_session

# create MaxCompute entrance object
o = ODPS(
    project='your-default-project',
    endpoint='your-end-point',
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)
# create maxframe session
session = new_session(o)

# perform data transformation
df = md.read_odps_table("test_source_table")
df["a"] = "prefix_" + df["a"]
md.to_odps_table(df, "test_prefix_source_table").execute()

# destroy maxframe session
session.destroy()

Overview

MaxFrame is a framework for large-scale data computation built on MaxCompute by Alibaba Cloud with API-compatibility for pandas. It intends to become an inplace replacement for Python users familiar with Numpy or Pandas APIs to utilize MaxCompute to run their code in a distributed environment.

10 minutes to MaxFrame

Here, movielens 100K is used as an example. Assume that three tables already exist, which are maxframe_ml_100k_movies (movie-related data), maxframe_ml_100k_users (user-related data), and maxframe_ml_100k_ratings (rating-related data).

Create a MaxFrame session object before starting the following steps:

import os
from odps import ODPS
from maxframe import new_session

# Make sure environment variable ODPS_ACCESS_ID already set to Access Key ID of user
# while environment variable ODPS_ACCESS_KEY set to Access Key Secret of user.
# Not recommended to hardcode Access Key ID or Access Key Secret in your code.
o = ODPS(
    access_id=os.getenv('ODPS_ACCESS_ID'),
    secret_access_key=os.getenv('ODPS_ACCESS_KEY'),
    project='**your-project**',
    endpoint='**your-endpoint**',
    user_agent='AlibabaCloud-Agent-Skills/alibabacloud-odps-maxframe-coding'
)
session = new_session(o)

You only need to use read_odps_table API to create a DataFrame object. For instance,

import maxframe.dataframe as md

users = md.read_odps_table('pyodps_ml_100k_users')

View columns of DataFrame and the types of the columns through the dtypes attribute, as shown in the following code:

>>> users.dtypes
user_id        int64
age            int64
sex           object
occupation    object
zip_code      object
dtype: object

Simply view the representation of the object will automatically show the first and last rows of the DataFrame.

>>> users
   user_id  age  sex     occupation  zip_code
0        1   24    M     technician     85711
1        2   53    F          other     94043
2        3   23    M         writer     32067
3        4   24    M     technician     43537
4        5   33    F          other     15213
...
5        6   42    M      executive     98101
6        7   57    M  administrator     91344
7        8   36    M  administrator     05201
8        9   29    M        student     01002
9       10   53    M         lawyer     90703

You can use the head method to obtain the first N data records for easy and quick data preview. For example:

>>> users.head(10).execute().fetch()
   user_id  age  sex     occupation  zip_code
0        1   24    M     technician     85711
1        2   53    F          other     94043
2        3   23    M         writer     32067
3        4   24    M     technician     43537
4        5   33    F          other     15213
5        6   42    M      executive     98101
6        7   57    M  administrator     91344
7        8   36    M  administrator     05201
8        9   29    M        student     01002
9       10   53    M         lawyer     90703

You can add a filter on the columns if you do not want to view all of them. For example:

>>> users[['user_id', 'age']].head(5).execute().fetch()
   user_id  age
0        1   24
1        2   53
2        3   23
3        4   24
4        5   33

You can also drop several columns. For example:

>>> users.drop(columns=['zip_code', 'age']).head(5)
   user_id  sex  occupation
0        1    M  technician
1        2    F       other
2        3    M      writer
3        4    M  technician
4        5    F       other

When excluding some columns, you may want to obtain new columns through computation. For example, add the sex_bool attribute and set it to True if sex is Male. Otherwise, set it to False. For example:

>>> users = users.drop(['zip_code', 'sex'])
>>> users["sex_bool"] = users.sex == "M"
>>> users.head(5).execute().fetch()
   user_id  age  occupation  sex_bool
0        1   24  technician      True
1        2   53       other     False
2        3   23      writer      True
3        4   24  technician      True
4        5   33       other     False

Obtain the number of persons at age of 20 to 25, as shown in the following code:

>>> users[users.age.between(20, 25)].count().execute().fetch()
195

Obtain the numbers of male and female users, as shown in the following code:

>>> users.groupby(users.sex).user_id.size()
F   273
M   670
dtype: int64

To divide users by job, obtain the first 10 jobs that have the largest population, and sort the jobs in the descending order of population. See the following:

>>> df = users.groupby("occupation").agg({"user_id": "count"})
>>> df.sort_values("user_id", ascending=False)[:10]
               user_id
occupation
student            196
other              105
educator            95
administrator       79
engineer            67
programmer          66
librarian           51
writer              45
executive           32
scientist           31

DataFrame APIs provide the value_counts method to quickly achieve the same result. An example is shown below.

>>> uses.occupation.value_counts()[:10]
student        196
other          105
educator        95
administrator   79
engineer        67
programmer      66
librarian       51
writer          45
executive       32
scientist       31
dtype: int64

Show data in a more intuitive graph, as shown in the following code:

%matplotlib inline

Use a horizontal bar chart to visualize data, as shown in the following code:

>>> users['occupation'].value_counts().plot(kind='barh', x='occupation', ylabel='prefession')
<matplotlib.axes._subplots.AxesSubplot at 0x10653cfd0>

\_images/df-value-count-plot.png

Divide ages into 30 groups and view the histogram of age distribution, as shown in the following code:

>>> users.age.hist(bins=30, title="Distribution of users' ages", xlabel='age', ylabel='count of users')
<matplotlib.axes._subplots.AxesSubplot at 0x10667a510>

\_images/df-age-hist.png

Use join to join the three tables and save the joined tables as a new table. For example:

>>> movies = md.read_odps_table('pyodps_ml_100k_movies')
>>> ratings = md.read_odps_table('pyodps_ml_100k_ratings')
>>>
>>> o.delete_table('pyodps_ml_100k_lens', if_exists=True)
>>> lens = movies.join(ratings).join(users).persist('pyodps_ml_100k_lens')
>>>
>>> lens.dtypes
odps.Schema {
movie_id                            int64
title                               string
release_date                        string
video_release_date                  string
imdb_url                            string
user_id                             int64
rating                              int64
unix_timestamp                      int64
age                                 int64
sex                                 string
occupation                          string
zip_code                            string
}

<!-- Divide ages of 0 to 80 into eight groups, as shown in the following code: --> <!-- labels = ['0-9', '10-19', '20-29', '30-39', '40-49', '50-59', '60-69', '70-79'] --> <!-- cut_lens = lens[lens, lens.age.cut(range(0, 81, 10), right=False, labels=labels).rename('age_group')] --> <!-- View the first 10 data records of a single age in a group, as shown in the following code: --> <!-- .. code-block:: python --> <!-- >>> cut_lens['age_group', 'age'].distinct()[:10] --> <!-- age_group age --> <!-- 0 0-9 7 --> <!-- 1 10-19 10 --> <!-- 2 10-19 11 --> <!-- 3 10-19 13 --> <!-- 4 10-19 14 --> <!-- 5 10-19 15 --> <!-- 6 10-19 16 --> <!-- 7 10-19 17 --> <!-- 8 10-19 18 --> <!-- 9 10-19 19 --> <!-- View users’ total rating and average rating of each age group, as shown in the following code: --> <!-- cut_lens.groupby('age_group').agg(cut_lens.rating.count().rename('total_rating'), cut_lens.rating.mean().rename('avg_rating')) --> <!-- age_group avg_rating total_rating --> <!-- 0 0-9 3.767442 43 --> <!-- 1 10-19 3.486126 8181 --> <!-- 2 20-29 3.467333 39535 --> <!-- 3 30-39 3.554444 25696 --> <!-- 4 40-49 3.591772 15021 --> <!-- 5 50-59 3.635800 8704 --> <!-- 6 60-69 3.648875 2623 --> <!-- 7 70-79 3.649746 197 -->

Getting started tutorials

• 10 minutes to MaxFrame

<a id="index"></a>

MaxFrame Documentation

MaxFrame is a framework for large-scale data computation built on MaxCompute by Alibaba Cloud with API-compatibility for pandas. It intends to become an inplace replacement for Python users familiar with Numpy or Pandas APIs to utilize MaxCompute to run their code in a distributed environment.

<a id="generated-dataframe"></a>

DataFrame

Constructor

Attributes and underlying data

Axes

Conversion

Indexing, iteration

Binary operator functions

Function application, GroupBy & window

<a id="generated-dataframe-stats"></a>

Computations / descriptive stats

Reindexing / selection / label manipulation

<a id="generated-dataframe-missing"></a>

Missing data handling

Reshaping, sorting, transposing

Combining / comparing / joining / merging

Time series-related

<a id="generated-dataframe-plotting"></a>

Plotting

DataFrame.plot is both a callable method and a namespace attribute for specific plotting methods of the form DataFrame.plot.<kind>.

<a id="generated-dataframe-io"></a>

Serialization / IO / conversion

<a id="generated-dataframe-mf"></a>

MaxFrame Extensions

DataFrame.mf provides methods unique to MaxFrame. These methods are collated from application scenarios in MaxCompute and these can be accessed like DataFrame.mf.<function/property>.

<a id="generated-general-functions"></a>

General functions

Data manipulations

Top-level missing data

Top-level dealing with numeric data

Top-level dealing with datetimelike

Top-level evaluation

maxframe.dataframe.DataFrame.abs

DataFrame.abs()

maxframe.dataframe.DataFrame.add_prefix

DataFrame.add_prefix(prefix)

Prefix labels with string prefix.

For Series, the row labels are prefixed. For DataFrame, the column labels are prefixed.

• Parameters:

prefix (*str*) – The string to add before each label.

• Returns:

New Series or DataFrame with updated labels.

• Return type:

Series or DataFrame

SEE ALSO

`Series.add_suffix` : Suffix row labels with string suffix.

`DataFrame.add_suffix` : Suffix column labels with string suffix.

Examples

>>> import maxframe.dataframe as md
>>> s = md.Series([1, 2, 3, 4])
>>> s.execute()
0    1
1    2
2    3
3    4
dtype: int64

>>> s.add_prefix('item_').execute()
item_0    1
item_1    2
item_2    3
item_3    4
dtype: int64

>>> df = md.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df.execute()
   A  B
0  1  3
1  2  4
2  3  5
3  4  6

>>> df.add_prefix('col_').execute()
     col_A  col_B
0        1      3
1        2      4
2        3      5
3        4      6

maxframe.dataframe.DataFrame.add_suffix

DataFrame.add_suffix(suffix)

Suffix labels with string suffix.

For Series, the row labels are suffixed. For DataFrame, the column labels are suffixed.

• Parameters:

suffix (*str*) – The string to add after each label.

• Returns:

New Series or DataFrame with updated labels.

• Return type:

Series or DataFrame

SEE ALSO

`Series.add_prefix` : Suffix row labels with string prefix.

`DataFrame.add_prefix` : Suffix column labels with string prefix.

Examples

>>> import maxframe.dataframe as md
>>> s = md.Series([1, 2, 3, 4])
>>> s.execute()
0    1
1    2
2    3
3    4
dtype: int64

>>> s.add_prefix('_item').execute()
0_item    1
1_item    2
2_item    3
3_item    4
dtype: int64

>>> df = md.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df.execute()
   A  B
0  1  3
1  2  4
2  3  5
3  4  6

>>> df.add_prefix('_col').execute()
     A_col  B_col
0        1      3
1        2      4
2        3      5
3        4      6

maxframe.dataframe.DataFrame.add

DataFrame.add(other, axis='columns', level=None, fill_value=None)

Get Addition of dataframe and other, element-wise (binary operator add). Equivalent to +, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, radd. Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, \, /, //, %, \\*.

• Parameters:
• other (scalar , sequence , *Series* , or *DataFrame*) – Any single or multiple element data structure, or list-like object.
• axis ( {0 or 'index' , 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns

(1 or ‘columns’). For Series input, axis to match Series index on.

• level (*int* or label) – Broadcast across a level, matching Index values on the

passed MultiIndex level.

• fill_value (*float* or None , default None) – Fill existing missing (NaN) values, and any new element needed for

successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.

• Returns:

Result of the arithmetic operation.

• Return type:

DataFrame

SEE ALSO

`DataFrame.add` : Add DataFrames.

`DataFrame.sub` : Subtract DataFrames.

`DataFrame.mul` : Multiply DataFrames.

`DataFrame.div` : Divide DataFrames (float division).

`DataFrame.truediv` : Divide DataFrames (float division).

`DataFrame.floordiv` : Divide DataFrames (integer division).

`DataFrame.mod` : Calculate modulo (remainder after division).

`DataFrame.pow` : Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

>>> import maxframe.dataframe as md
>>> df = md.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df.execute()
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same results.

>>> (df + 1).execute()
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1).execute()
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10).execute()
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10).execute()
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> (df - [1, 2]).execute()
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns').execute()
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(md.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index').execute()
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = md.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other.execute()
           angles
circle          0
triangle        3
rectangle       4

>>> df.mul(other, fill_value=0).execute()
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

maxframe.dataframe.DataFrame.all

DataFrame.all(axis=0, bool_only=None, skipna=True, level=None, method=None)

maxframe.dataframe.DataFrame.any

DataFrame.any(axis=0, bool_only=None, skipna=True, level=None, method=None)

maxframe.dataframe.DataFrame.columns

property DataFrame.columns

maxframe.dataframe.DataFrame.copy

DataFrame.copy() → TileableType

maxframe.dataframe.DataFrame.count

DataFrame.count(axis=0, level=None, numeric_only=False, \\kw)

maxframe.dataframe.DataFrame.index

property DataFrame.index

maxframe.dataframe.DataFrame.max

DataFrame.max(axis=None, skipna=True, level=None, numeric_only=None, method=None)

maxframe.dataframe.DataFrame.mean

DataFrame.mean(axis=None, skipna=True, level=None, numeric_only=None, method=None)

maxframe.dataframe.DataFrame.median

DataFrame.median(axis=0, skipna=True, level=None, numeric_only=None, method=None)