1. 多线程编程
1.1 threading模块基础
01.模块导入与基本概念
a.threading模块简介
threading模块是Python标准库中用于多线程编程的核心模块，提供了线程创建、同步、通信等功能。相比底层的thread模块，threading提供了更高级的面向对象接口，支持守护线程、线程本地数据等特性。
b.导入示例
---
import threading
import time
# 获取当前线程信息
current = threading.current_thread()
print(f"当前线程: {current.name}, ID: {current.ident}")
# 获取活动线程数量
active_count = threading.active_count()
print(f"活动线程数: {active_count}")
# 枚举所有活动线程
for thread in threading.enumerate():
print(f"线程: {thread.name}, 守护线程: {thread.daemon}")
---
02.线程对象属性
a.name属性
线程名称，用于标识线程，默认为Thread-N格式，可通过构造函数或name属性设置自定义名称。
b.ident属性
线程标识符，由操作系统分配的唯一整数，线程启动后才有值，线程结束后可能被重用。
c.daemon属性
守护线程标志，设置为True时线程作为守护线程运行，主线程退出时守护线程自动终止。
d.属性示例
---
def worker():
print(f"工作线程: {threading.current_thread().name}")
time.sleep(2)
# 创建线程并设置属性
t = threading.Thread(target=worker, name="Worker-1")
t.daemon = True # 设置为守护线程
print(f"启动前 - 名称: {t.name}, ID: {t.ident}, 守护: {t.daemon}")
t.start()
print(f"启动后 - 名称: {t.name}, ID: {t.ident}, 守护: {t.daemon}")
t.join() # 等待线程结束
---
03.线程状态管理
a.is_alive方法
检查线程是否处于活动状态，从start调用到run方法结束期间返回True。
b.join方法
阻塞调用线程，等待目标线程结束，可指定超时时间，超时后继续执行但线程可能仍在运行。
c.状态管理示例
---
import threading
import time
def long_task(duration):
print(f"{threading.current_thread().name} 开始执行")
time.sleep(duration)
print(f"{threading.current_thread().name} 执行完成")
# 创建并启动线程
t1 = threading.Thread(target=long_task, args=(3,), name="Task-1")
t2 = threading.Thread(target=long_task, args=(5,), name="Task-2")
t1.start()
t2.start()
# 检查线程状态
print(f"t1 存活: {t1.is_alive()}, t2 存活: {t2.is_alive()}")
# 等待t1完成，超时2秒
t1.join(timeout=2)
print(f"等待2秒后 t1 存活: {t1.is_alive()}")
# 等待所有线程完成
t1.join()
t2.join()
print("所有线程已完成")
---
1.2 Thread类与线程创建
01.使用函数创建线程
a.target参数
Thread构造函数的target参数接受可调用对象，线程启动时执行该函数，通过args和kwargs传递参数。
b.函数创建示例
---
import threading
import time
def download_file(filename, size):
print(f"开始下载 {filename}, 大小: {size}MB")
time.sleep(2) # 模拟下载
print(f"{filename} 下载完成")
# 创建多个线程
threads = []
files = [("file1.zip", 100), ("file2.zip", 200), ("file3.zip", 150)]
for filename, size in files:
t = threading.Thread(target=download_file, args=(filename, size))
threads.append(t)
t.start()
# 等待所有下载完成
for t in threads:
t.join()
print("所有文件下载完成")
---
02.继承Thread类创建线程
a.重写run方法
继承Thread类并重写run方法，将线程执行逻辑放在run方法中，通过start方法启动线程。
b.继承方式示例
---
import threading
import time
import random
class WorkerThread(threading.Thread):
def __init__(self, name, task_count):
super().__init__(name=name)
self.task_count = task_count
self.results = []
def run(self):
print(f"{self.name} 开始执行 {self.task_count} 个任务")
for i in range(self.task_count):
time.sleep(random.uniform(0.1, 0.5))
result = f"任务{i+1}完成"
self.results.append(result)
print(f"{self.name}: {result}")
print(f"{self.name} 所有任务完成")
# 创建并启动线程
worker1 = WorkerThread("Worker-1", 3)
worker2 = WorkerThread("Worker-2", 5)
worker1.start()
worker2.start()
worker1.join()
worker2.join()
print(f"Worker-1 结果: {worker1.results}")
print(f"Worker-2 结果: {worker2.results}")
---
03.线程参数传递
a.位置参数与关键字参数
通过args传递位置参数元组，通过kwargs传递关键字参数字典，支持混合使用。
b.参数传递示例
---
import threading
def process_data(data, prefix="处理", suffix="完成", delay=1):
import time
print(f"{prefix} {data}")
time.sleep(delay)
print(f"{data} {suffix}")
# 使用位置参数
t1 = threading.Thread(target=process_data, args=("数据A",))
t1.start()
# 使用关键字参数
t2 = threading.Thread(
target=process_data,
args=("数据B",),
kwargs={"prefix": "开始", "suffix": "结束", "delay": 0.5}
)
t2.start()
# 混合使用
t3 = threading.Thread(
target=process_data,
args=("数据C",),
kwargs={"delay": 1.5}
)
t3.start()
t1.join()
t2.join()
t3.join()
---
1.3 线程同步与锁
01.Lock互斥锁
a.锁的基本概念
Lock是最基本的同步原语，同一时刻只有一个线程可以获取锁，其他线程必须等待锁释放。使用acquire获取锁，release释放锁，推荐使用with语句自动管理。
b.Lock使用示例
---
import threading
import time
# 共享资源
counter = 0
lock = threading.Lock()
def increment(n):
global counter
for _ in range(n):
# 不使用锁会导致竞态条件
with lock: # 自动获取和释放锁
temp = counter
time.sleep(0.0001) # 模拟耗时操作
counter = temp + 1
# 创建多个线程
threads = []
for i in range(5):
t = threading.Thread(target=increment, args=(1000,))
threads.append(t)
t.start()
for t in threads:
t.join()
print(f"最终计数: {counter}") # 应该是5000
---
02.RLock可重入锁
a.可重入特性
RLock允许同一线程多次获取同一把锁，必须释放相同次数才能完全解锁。适用于递归函数或需要多次获取锁的场景。
b.RLock使用示例
---
import threading
class BankAccount:
def __init__(self, balance):
self.balance = balance
self.lock = threading.RLock()
def deposit(self, amount):
with self.lock:
print(f"存款 {amount}")
self.balance += amount
self._log_transaction("存款", amount)
def withdraw(self, amount):
with self.lock:
if self.balance >= amount:
print(f"取款 {amount}")
self.balance -= amount
self._log_transaction("取款", amount)
return True
return False
def _log_transaction(self, type, amount):
# 内部方法也需要获取锁，使用RLock避免死锁
with self.lock:
print(f"记录: {type} {amount}, 余额: {self.balance}")
account = BankAccount(1000)
def do_transactions():
account.deposit(500)
account.withdraw(200)
t1 = threading.Thread(target=do_transactions)
t2 = threading.Thread(target=do_transactions)
t1.start()
t2.start()
t1.join()
t2.join()
print(f"最终余额: {account.balance}")
---
03.死锁问题与避免
a.死锁产生条件
多个线程相互等待对方持有的锁，形成循环等待。常见于多个锁的获取顺序不一致，或嵌套锁使用不当。
b.死锁避免示例
---
import threading
import time
lock_a = threading.Lock()
lock_b = threading.Lock()
def task1():
# 错误方式：可能导致死锁
# with lock_a:
# time.sleep(0.1)
# with lock_b:
# print("Task1 完成")
# 正确方式：统一锁的获取顺序
locks = sorted([lock_a, lock_b], key=id)
with locks[0]:
time.sleep(0.1)
with locks[1]:
print("Task1 完成")
def task2():
# 统一顺序避免死锁
locks = sorted([lock_a, lock_b], key=id)
with locks[0]:
time.sleep(0.1)
with locks[1]:
print("Task2 完成")
t1 = threading.Thread(target=task1)
t2 = threading.Thread(target=task2)
t1.start()
t2.start()
t1.join()
t2.join()
print("所有任务完成，无死锁")
---
1.4 线程间通信
01.Queue队列通信
a.队列基本操作
Queue提供线程安全的FIFO队列，支持put放入数据、get获取数据、task_done标记任务完成、join等待所有任务完成。
b.Queue使用示例
---
import threading
import queue
import time
import random
# 创建队列
task_queue = queue.Queue()
result_queue = queue.Queue()
def producer(n):
for i in range(n):
task = f"任务-{i}"
task_queue.put(task)
print(f"生产者: 添加 {task}")
time.sleep(random.uniform(0.1, 0.3))
print("生产者: 完成")
def consumer(name):
while True:
try:
task = task_queue.get(timeout=2)
print(f"{name}: 处理 {task}")
time.sleep(random.uniform(0.2, 0.5))
result_queue.put(f"{task}-完成")
task_queue.task_done()
except queue.Empty:
print(f"{name}: 队列为空，退出")
break
# 启动生产者
p = threading.Thread(target=producer, args=(10,))
p.start()
# 启动多个消费者
consumers = []
for i in range(3):
c = threading.Thread(target=consumer, args=(f"消费者-{i}",))
consumers.append(c)
c.start()
p.join()
task_queue.join() # 等待所有任务完成
print(f"处理结果数: {result_queue.qsize()}")
---
02.Event事件通信
a.Event机制
Event用于线程间的简单信号通信，一个线程发送信号，其他线程等待信号。支持set设置信号、clear清除信号、wait等待信号、is_set检查状态。
b.Event使用示例
---
import threading
import time
# 创建事件对象
start_event = threading.Event()
stop_event = threading.Event()
def worker(name):
print(f"{name}: 等待开始信号")
start_event.wait() # 阻塞直到事件被设置
print(f"{name}: 开始工作")
while not stop_event.is_set():
print(f"{name}: 工作中...")
time.sleep(1)
print(f"{name}: 收到停止信号，退出")
# 创建工作线程
threads = []
for i in range(3):
t = threading.Thread(target=worker, args=(f"Worker-{i}",))
threads.append(t)
t.start()
time.sleep(2)
print("主线程: 发送开始信号")
start_event.set() # 设置事件，唤醒所有等待线程
time.sleep(5)
print("主线程: 发送停止信号")
stop_event.set() # 设置停止事件
for t in threads:
t.join()
print("所有线程已结束")
---
03.Condition条件变量
a.条件变量机制
Condition结合锁和事件，允许线程等待特定条件满足。支持wait等待条件、notify唤醒一个等待线程、notify_all唤醒所有等待线程。
b.Condition使用示例
---
import threading
import time
class SharedBuffer:
def __init__(self, size):
self.buffer = []
self.size = size
self.condition = threading.Condition()
def produce(self, item):
with self.condition:
while len(self.buffer) >= self.size:
print("缓冲区满，生产者等待")
self.condition.wait() # 等待消费者消费
self.buffer.append(item)
print(f"生产: {item}, 缓冲区: {len(self.buffer)}")
self.condition.notify() # 通知消费者
def consume(self):
with self.condition:
while len(self.buffer) == 0:
print("缓冲区空，消费者等待")
self.condition.wait() # 等待生产者生产
item = self.buffer.pop(0)
print(f"消费: {item}, 缓冲区: {len(self.buffer)}")
self.condition.notify() # 通知生产者
return item
buffer = SharedBuffer(5)
def producer():
for i in range(10):
buffer.produce(f"数据-{i}")
time.sleep(0.1)
def consumer():
for _ in range(10):
buffer.consume()
time.sleep(0.2)
p = threading.Thread(target=producer)
c = threading.Thread(target=consumer)
p.start()
c.start()
p.join()
c.join()
---
1.5 线程池ThreadPoolExecutor
01.线程池基本使用
a.线程池概念
ThreadPoolExecutor管理线程池，复用线程执行任务，避免频繁创建销毁线程的开销。通过submit提交任务返回Future对象，通过map批量处理任务。
b.基本使用示例
---
from concurrent.futures import ThreadPoolExecutor
import time
def task(n):
print(f"任务 {n} 开始")
time.sleep(1)
result = n * n
print(f"任务 {n} 完成，结果: {result}")
return result
# 创建线程池，最多3个工作线程
with ThreadPoolExecutor(max_workers=3) as executor:
# 提交单个任务
future1 = executor.submit(task, 5)
print(f"任务已提交，Future对象: {future1}")
# 提交多个任务
futures = [executor.submit(task, i) for i in range(1, 6)]
# 获取结果
for future in futures:
result = future.result() # 阻塞等待结果
print(f"获取结果: {result}")
print("所有任务完成")
---
02.map方法批量处理
a.map方法特点
map方法接受函数和可迭代对象，按顺序返回结果，自动管理任务提交和结果收集，比手动submit更简洁。
b.map使用示例
---
from concurrent.futures import ThreadPoolExecutor
import time
import requests
def download_url(url):
print(f"下载: {url}")
try:
response = requests.get(url, timeout=5)
size = len(response.content)
print(f"完成: {url}, 大小: {size} 字节")
return url, size
except Exception as e:
print(f"失败: {url}, 错误: {e}")
return url, 0
urls = [
"https://www.python.org",
"https://www.github.com",
"https://www.stackoverflow.com",
"https://www.reddit.com",
"https://www.wikipedia.org"
]
# 使用map批量下载
with ThreadPoolExecutor(max_workers=3) as executor:
results = executor.map(download_url, urls)
# 按提交顺序获取结果
for url, size in results:
print(f"结果: {url} -> {size} 字节")
print("批量下载完成")
---
03.线程池参数配置
a.参数说明
max_workers指定最大线程数，thread_name_prefix设置线程名称前缀，initializer指定线程初始化函数，initargs传递初始化参数。
b.参数配置示例
---
from concurrent.futures import ThreadPoolExecutor
import threading
import time
# 线程本地存储
thread_local = threading.local()
def init_worker(db_config):
# 每个线程初始化时调用
thread_local.db_conn = f"数据库连接-{threading.current_thread().name}"
thread_local.config = db_config
print(f"{threading.current_thread().name} 初始化完成")
def process_record(record_id):
# 使用线程本地存储
conn = thread_local.db_conn
config = thread_local.config
print(f"处理记录 {record_id}, 连接: {conn}, 配置: {config}")
time.sleep(0.5)
return f"记录-{record_id}-已处理"
# 配置线程池
with ThreadPoolExecutor(
max_workers=4,
thread_name_prefix="Worker",
initializer=init_worker,
initargs=({"host": "localhost", "port": 5432},)
) as executor:
# 提交任务
futures = [executor.submit(process_record, i) for i in range(10)]
# 获取结果
for future in futures:
print(future.result())
print("所有记录处理完成")
---
2. 多进程编程
2.1 multiprocessing模块
01.模块基础与特性
a.multiprocessing简介
multiprocessing模块提供跨平台的多进程支持，每个进程拥有独立的Python解释器和内存空间，绕过GIL限制实现真正的并行计算。模块API设计类似threading，降低学习成本。
b.基础使用示例
---
import multiprocessing
import os
import time
def worker(name, duration):
print(f"进程 {name} 启动, PID: {os.getpid()}, 父进程: {os.getppid()}")
time.sleep(duration)
print(f"进程 {name} 完成")
return f"{name}-结果"
if __name__ == '__main__':
print(f"主进程 PID: {os.getpid()}")
print(f"CPU核心数: {multiprocessing.cpu_count()}")
# 创建进程
p1 = multiprocessing.Process(target=worker, args=("Worker-1", 2))
p2 = multiprocessing.Process(target=worker, args=("Worker-2", 1))
# 启动进程
p1.start()
p2.start()
# 等待进程完成
p1.join()
p2.join()
print("所有进程已完成")
---
02.进程属性与方法
a.进程对象属性
name进程名称，pid进程ID，daemon守护进程标志，exitcode退出码。进程启动后pid才有值，正常退出exitcode为0，异常退出为非0值。
b.进程方法
start启动进程，join等待进程结束，is_alive检查进程状态，terminate强制终止进程，kill发送SIGKILL信号。
c.属性方法示例
---
import multiprocessing
import time
import sys
def long_running_task(n):
try:
for i in range(n):
print(f"进程 {multiprocessing.current_process().name} 执行步骤 {i+1}/{n}")
time.sleep(1)
return 0
except KeyboardInterrupt:
print(f"进程 {multiprocessing.current_process().name} 被中断")
return 1
if __name__ == '__main__':
p = multiprocessing.Process(
target=long_running_task,
args=(10,),
name="LongTask"
)
print(f"启动前 - 名称: {p.name}, PID: {p.pid}, 存活: {p.is_alive()}")
p.start()
print(f"启动后 - 名称: {p.name}, PID: {p.pid}, 存活: {p.is_alive()}")
# 等待3秒后终止
time.sleep(3)
if p.is_alive():
print(f"终止进程 {p.name}")
p.terminate()
p.join(timeout=2)
print(f"进程退出码: {p.exitcode}")
print(f"进程状态: {'存活' if p.is_alive() else '已结束'}")
---
03.守护进程
a.守护进程特性
守护进程随主进程退出而自动终止，不能创建子进程。适用于后台服务任务，必须在start之前设置daemon属性。
b.守护进程示例
---
import multiprocessing
import time
def daemon_task():
print(f"守护进程启动, PID: {multiprocessing.current_process().pid}")
try:
while True:
print("守护进程运行中...")
time.sleep(1)
except KeyboardInterrupt:
print("守护进程被中断")
def normal_task(duration):
print(f"普通进程启动, PID: {multiprocessing.current_process().pid}")
for i in range(duration):
print(f"普通进程工作中 {i+1}/{duration}")
time.sleep(1)
print("普通进程完成")
if __name__ == '__main__':
# 创建守护进程
daemon_p = multiprocessing.Process(target=daemon_task, name="Daemon")
daemon_p.daemon = True
# 创建普通进程
normal_p = multiprocessing.Process(target=normal_task, args=(3,), name="Normal")
daemon_p.start()
normal_p.start()
print(f"守护进程: {daemon_p.name}, daemon={daemon_p.daemon}")
print(f"普通进程: {normal_p.name}, daemon={normal_p.daemon}")
# 只等待普通进程
normal_p.join()
print("主进程退出，守护进程将自动终止")
---
2.2 Process类与进程创建
01.使用函数创建进程
a.target方式创建
通过target参数传递函数对象，args传递位置参数，kwargs传递关键字参数。进程间数据隔离，修改不影响主进程。
b.函数创建示例
---
import multiprocessing
import time
import math
def compute_factorial(n, process_name):
print(f"{process_name} 开始计算 {n}!")
result = math.factorial(n)
print(f"{process_name} 计算完成: {n}! = {result}")
return result
def compute_prime(start, end, process_name):
print(f"{process_name} 查找 {start}-{end} 之间的质数")
primes = []
for num in range(start, end + 1):
if num > 1:
for i in range(2, int(math.sqrt(num)) + 1):
if num % i == 0:
break
else:
primes.append(num)
print(f"{process_name} 找到 {len(primes)} 个质数")
return primes
if __name__ == '__main__':
# 创建多个计算进程
processes = []
p1 = multiprocessing.Process(
target=compute_factorial,
args=(1000, "Factorial-Process")
)
processes.append(p1)
p2 = multiprocessing.Process(
target=compute_prime,
args=(1, 10000, "Prime-Process")
)
processes.append(p2)
# 启动所有进程
start_time = time.time()
for p in processes:
p.start()
# 等待所��进程完成
for p in processes:
p.join()
elapsed = time.time() - start_time
print(f"所有计算完成，耗时: {elapsed:.2f}秒")
---
02.继承Process类创建进程
a.自定义Process子类
继承Process类并重写run方法，可在__init__中初始化进程状态，通过实例变量传递数据。适合复杂进程逻辑封装。
b.继承方式示例
---
import multiprocessing
import time
import random
class DataProcessor(multiprocessing.Process):
def __init__(self, data_id, chunk_size):
super().__init__()
self.data_id = data_id
self.chunk_size = chunk_size
self.processed_count = 0
def run(self):
print(f"进程 {self.name} (PID: {self.pid}) 开始处理数据块 {self.data_id}")
for i in range(self.chunk_size):
# 模拟数据处理
time.sleep(random.uniform(0.01, 0.05))
self.processed_count += 1
if (i + 1) % 10 == 0:
print(f"{self.name}: 已处理 {i+1}/{self.chunk_size} 条数据")
print(f"{self.name} 完成，共处理 {self.processed_count} 条数据")
class MonitorProcess(multiprocessing.Process):
def __init__(self, check_interval):
super().__init__(name="Monitor")
self.check_interval = check_interval
self.daemon = True
def run(self):
print(f"监控进程启动 (PID: {self.pid})")
while True:
active = multiprocessing.active_children()
print(f"[监控] 当前活动进程数: {len(active)}")
time.sleep(self.check_interval)
if __name__ == '__main__':
# 启动监控进程
monitor = MonitorProcess(check_interval=2)
monitor.start()
# 创建数据处理进程
processors = []
for i in range(4):
p = DataProcessor(data_id=f"CHUNK-{i}", chunk_size=30)
processors.append(p)
p.start()
# 等待所有处理进程完成
for p in processors:
p.join()
print("所有数据处理完成")
---
03.进程启动方法
a.启动方法类型
spawn创建全新进程，fork复制父进程，forkserver从服务进程fork。Windows仅支持spawn，Unix默认fork，可通过set_start_method设置。
b.启动方法示例
---
import multiprocessing
import sys
import os
# 全局变量
global_counter = 0
def worker(name):
global global_counter
print(f"{name} 启动, PID: {os.getpid()}")
print(f"{name} 看到的 global_counter: {global_counter}")
# 修改全局变量
global_counter += 10
print(f"{name} 修改后 global_counter: {global_counter}")
if __name__ == '__main__':
# 获取当前启动方法
print(f"当前启动方法: {multiprocessing.get_start_method()}")
print(f"支持的启动方法: {multiprocessing.get_all_start_methods()}")
# 设置启动方法（必须在创建进程前）
# multiprocessing.set_start_method('spawn') # 强制使用spawn
print(f"主进程 PID: {os.getpid()}")
print(f"主进程 global_counter: {global_counter}")
# 创建进程
p1 = multiprocessing.Process(target=worker, args=("Worker-1",))
p2 = multiprocessing.Process(target=worker, args=("Worker-2",))
p1.start()
p2.start()
p1.join()
p2.join()
# spawn方式下，子进程修改不影响主进程
print(f"主进程最终 global_counter: {global_counter}")
---
2.3 进程间通信IPC
01.Queue队列通信
a.Queue特性
multiprocessing.Queue基于管道和锁实现，支持多进程安全的FIFO队列。提供put放入数据、get获取数据、qsize查询大小、empty和full检查状态。
b.Queue使用示例
---
import multiprocessing
import time
import random
def producer(queue, n, name):
print(f"生产者 {name} 启动")
for i in range(n):
item = f"{name}-数据-{i}"
queue.put(item)
print(f"{name} 生产: {item}")
time.sleep(random.uniform(0.1, 0.3))
print(f"{name} 生产完成")
def consumer(queue, name):
print(f"消费者 {name} 启动")
while True:
try:
# 设置超时避免永久阻塞
item = queue.get(timeout=2)
print(f"{name} 消费: {item}")
time.sleep(random.uniform(0.2, 0.4))
except multiprocessing.queues.Empty:
print(f"{name} 队列为空，退出")
break
if __name__ == '__main__':
# 创建队列，最大容量10
queue = multiprocessing.Queue(maxsize=10)
# 创建生产者进程
producers = [
multiprocessing.Process(target=producer, args=(queue, 5, "P1")),
multiprocessing.Process(target=producer, args=(queue, 5, "P2"))
]
# 创建消费者进程
consumers = [
multiprocessing.Process(target=consumer, args=(queue, "C1")),
multiprocessing.Process(target=consumer, args=(queue, "C2")),
multiprocessing.Process(target=consumer, args=(queue, "C3"))
]
# 启动所有进程
for p in producers + consumers:
p.start()
# 等待所有进程完成
for p in producers + consumers:
p.join()
print(f"最终队列大小: {queue.qsize()}")
---
02.Pipe管道通信
a.Pipe特性
Pipe创建双向管道，返回两个连接对象，支持send发送数据、recv接收数据。比Queue轻量但不支持多进程安全，适合两个进程间通信。
b.Pipe使用示例
---
import multiprocessing
import time
def sender(conn, messages):
print(f"发送进程启动, PID: {multiprocessing.current_process().pid}")
for msg in messages:
print(f"发送: {msg}")
conn.send(msg)
time.sleep(0.5)
# 发送结束信号
conn.send(None)
conn.close()
print("发送进程完成")
def receiver(conn):
print(f"接收进程启动, PID: {multiprocessing.current_process().pid}")
while True:
msg = conn.recv()
if msg is None:
print("收到结束信号")
break
print(f"接收: {msg}")
# 发送确认
conn.send(f"ACK: {msg}")
conn.close()
print("接收进程完成")
def bidirectional_comm(conn, name):
print(f"{name} 启动")
# 发送消息
for i in range(3):
msg = f"{name}-消息-{i}"
conn.send(msg)
print(f"{name} 发送: {msg}")
# 接收响应
response = conn.recv()
print(f"{name} 收到: {response}")
time.sleep(0.3)
conn.close()
if __name__ == '__main__':
# 单向通信示例
parent_conn, child_conn = multiprocessing.Pipe()
messages = ["Hello", "World", "From", "Parent"]
p = multiprocessing.Process(target=sender, args=(child_conn, messages))
p.start()
# 父进程接收消息
while True:
msg = parent_conn.recv()
if msg is None:
break
print(f"父进程收到: {msg}")
p.join()
parent_conn.close()
print("\n--- 双向通信示例 ---")
# 双向通信示例
conn1, conn2 = multiprocessing.Pipe()
p1 = multiprocessing.Process(target=bidirectional_comm, args=(conn1, "进程A"))
p2 = multiprocessing.Process(target=bidirectional_comm, args=(conn2, "进程B"))
p1.start()
p2.start()
p1.join()
p2.join()
---
03.Value和Array共享内存
a.共享内存特性
Value创建共享变量，Array创建共享数组，支持基本类型数据共享。使用锁保证线程安全，避免竞态条件。
b.共享内存示例
---
import multiprocessing
import time
def increment_counter(counter, lock, n):
for _ in range(n):
with lock:
counter.value += 1
time.sleep(0.001)
def update_array(shared_array, lock, start, end, value):
print(f"进程更新数组 [{start}:{end}] = {value}")
with lock:
for i in range(start, end):
shared_array[i] = value
time.sleep(0.01)
if __name__ == '__main__':
# Value共享变量示例
counter = multiprocessing.Value('i', 0) # 'i'表示整数
lock = multiprocessing.Lock()
processes = []
for i in range(5):
p = multiprocessing.Process(
target=increment_counter,
args=(counter, lock, 100)
)
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"最终计数: {counter.value}") # 应该是500
# Array共享数组示例
shared_array = multiprocessing.Array('d', 20) # 'd'表示double
array_lock = multiprocessing.Lock()
# 初始化数组
for i in range(len(shared_array)):
shared_array[i] = 0.0
# 多个进程更新不同区域
p1 = multiprocessing.Process(
target=update_array,
args=(shared_array, array_lock, 0, 5, 1.1)
)
p2 = multiprocessing.Process(
target=update_array,
args=(shared_array, array_lock, 5, 10, 2.2)
)
p3 = multiprocessing.Process(
target=update_array,
args=(shared_array, array_lock, 10, 15, 3.3)
)
p4 = multiprocessing.Process(
target=update_array,
args=(shared_array, array_lock, 15, 20, 4.4)
)
p1.start()
p2.start()
p3.start()
p4.start()
p1.join()
p2.join()
p3.join()
p4.join()
print(f"最终数组: {list(shared_array)}")
---
2.4 共享内存与Manager
01.Manager管理器
a.Manager特性
Manager提供服务进程管理共享对象，支持list、dict、Namespace等复杂数据结构。通过代理对象访问，自动处理进程间同步，但性能低于Value和Array。
b.Manager使用示例
---
import multiprocessing
import time
def worker_list(shared_list, worker_id):
for i in range(5):
item = f"Worker-{worker_id}-Item-{i}"
shared_list.append(item)
print(f"Worker {worker_id} 添加: {item}")
time.sleep(0.1)
def worker_dict(shared_dict, worker_id):
for i in range(3):
key = f"key_{worker_id}_{i}"
value = f"value_{worker_id}_{i}"
shared_dict[key] = value
print(f"Worker {worker_id} 设置: {key} = {value}")
time.sleep(0.15)
def worker_namespace(ns, worker_id):
# 读取共享命名空间
print(f"Worker {worker_id} 读取: counter = {ns.counter}")
# 修改需要重新赋值
ns.counter += 1
ns.data.append(f"data-{worker_id}")
print(f"Worker {worker_id} 修改后: counter = {ns.counter}")
if __name__ == '__main__':
# 创建Manager
with multiprocessing.Manager() as manager:
# 共享列表
shared_list = manager.list()
processes = []
for i in range(3):
p = multiprocessing.Process(target=worker_list, args=(shared_list, i))
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"\n共享列表最终内容 ({len(shared_list)} 项):")
for item in shared_list:
print(f" - {item}")
# 共享字典
shared_dict = manager.dict()
processes = []
for i in range(3):
p = multiprocessing.Process(target=worker_dict, args=(shared_dict, i))
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"\n共享字典最终内容:")
for key, value in shared_dict.items():
print(f" {key}: {value}")
# 共享命名空间
ns = manager.Namespace()
ns.counter = 0
ns.data = manager.list()
processes = []
for i in range(5):
p = multiprocessing.Process(target=worker_namespace, args=(ns, i))
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"\n命名空间最终状态:")
print(f" counter: {ns.counter}")
print(f" data: {list(ns.data)}")
---
02.SharedMemory共享内存
a.SharedMemory特性
Python3.8引入的SharedMemory模块，提供跨进程的原始内存块共享，性能优于Manager。适合大数据量传输，需手动管理内存生命周期。
b.SharedMemory使用示例
---
import multiprocessing
from multiprocessing import shared_memory
import numpy as np
import time
def worker_write(shm_name, shape, worker_id, start, end):
# 连接到已存在的共享内存
existing_shm = shared_memory.SharedMemory(name=shm_name)
# 创建numpy数组视图
arr = np.ndarray(shape, dtype=np.float64, buffer=existing_shm.buf)
# 写入数据
print(f"Worker {worker_id} 写入 [{start}:{end}]")
for i in range(start, end):
arr[i] = worker_id * 100 + i
time.sleep(0.01)
print(f"Worker {worker_id} 完成")
existing_shm.close()
def worker_read(shm_name, shape):
# 连接到共享内存
existing_shm = shared_memory.SharedMemory(name=shm_name)
# 读取数据
arr = np.ndarray(shape, dtype=np.float64, buffer=existing_shm.buf)
print(f"读取进程: 数组前10个元素: {arr[:10]}")
print(f"读取进程: 数组和: {np.sum(arr)}")
existing_shm.close()
if __name__ == '__main__':
# 创建共享内存
size = 1000
shm = shared_memory.SharedMemory(create=True, size=size * 8) # 8字节per float64
try:
# 创建numpy数组
arr = np.ndarray((size,), dtype=np.float64, buffer=shm.buf)
arr[:] = 0.0 # 初始化
print(f"创建共享内存: {shm.name}, 大小: {shm.size} 字节")
# 创建多个写入进程
processes = []
chunk_size = size // 4
for i in range(4):
start = i * chunk_size
end = start + chunk_size if i < 3 else size
p = multiprocessing.Process(
target=worker_write,
args=(shm.name, (size,), i, start, end)
)
processes.append(p)
p.start()
# 等待写入完成
for p in processes:
p.join()
# 主进程读取结果
print(f"\n主进程读取: 前20个元素: {arr[:20]}")
print(f"主进程读取: 后20个元素: {arr[-20:]}")
# 创建读取进程
reader = multiprocessing.Process(
target=worker_read,
args=(shm.name, (size,))
)
reader.start()
reader.join()
finally:
# 清理共享内存
shm.close()
shm.unlink()
print("\n共享内存已清理")
---
03.Manager性能优化
a.性能对比
Value和Array直接操作共享内存性能最高，Manager通过代理对象性能较低，SharedMemory适合大数据传输。选择时需权衡易用性和性能。
b.性能测试示例
---
import multiprocessing
from multiprocessing import shared_memory
import time
import numpy as np
def test_value(counter, lock, iterations):
for _ in range(iterations):
with lock:
counter.value += 1
def test_manager_dict(shared_dict, worker_id, iterations):
key = f"worker_{worker_id}"
for i in range(iterations):
shared_dict[key] = i
def test_shared_memory(shm_name, index, iterations):
shm = shared_memory.SharedMemory(name=shm_name)
arr = np.ndarray((1000,), dtype=np.int32, buffer=shm.buf)
for i in range(iterations):
arr[index] = i
shm.close()
if __name__ == '__main__':
iterations = 1000
num_workers = 4
# 测试Value性能
print("测试 Value 性能...")
counter = multiprocessing.Value('i', 0)
lock = multiprocessing.Lock()
start = time.time()
processes = []
for i in range(num_workers):
p = multiprocessing.Process(target=test_value, args=(counter, lock, iterations))
processes.append(p)
p.start()
for p in processes:
p.join()
value_time = time.time() - start
print(f"Value 耗时: {value_time:.3f}秒, 最终值: {counter.value}")
# 测试Manager性能
print("\n测试 Manager 性能...")
with multiprocessing.Manager() as manager:
shared_dict = manager.dict()
start = time.time()
processes = []
for i in range(num_workers):
p = multiprocessing.Process(
target=test_manager_dict,
args=(shared_dict, i, iterations)
)
processes.append(p)
p.start()
for p in processes:
p.join()
manager_time = time.time() - start
print(f"Manager 耗时: {manager_time:.3f}秒, 字典大小: {len(shared_dict)}")
# 测试SharedMemory性能
print("\n测试 SharedMemory 性能...")
shm = shared_memory.SharedMemory(create=True, size=1000 * 4)
try:
start = time.time()
processes = []
for i in range(num_workers):
p = multiprocessing.Process(
target=test_shared_memory,
args=(shm.name, i, iterations)
)
processes.append(p)
p.start()
for p in processes:
p.join()
shm_time = time.time() - start
print(f"SharedMemory 耗时: {shm_time:.3f}秒")
finally:
shm.close()
shm.unlink()
# 性能对比
print(f"\n性能对比:")
print(f" Value: {value_time:.3f}秒 (基准)")
print(f" Manager: {manager_time:.3f}秒 ({manager_time/value_time:.2f}x)")
print(f" SharedMemory: {shm_time:.3f}秒 ({shm_time/value_time:.2f}x)")
---
2.5 进程池ProcessPoolExecutor
01.进程池基本使用
a.进程池概念
ProcessPoolExecutor管理进程池，复用进程执行CPU密集型任务，避免进程创建销毁开销。默认进程数为CPU核心数，通过submit提交任务，map批量处理。
b.基本使用示例
---
from concurrent.futures import ProcessPoolExecutor
import time
import os
def cpu_bound_task(n):
pid = os.getpid()
print(f"进程 {pid} 开始计算 {n}")
# CPU密集型计算
result = 0
for i in range(n):
result += i ** 2
print(f"进程 {pid} 完成，结果: {result}")
return result
if __name__ == '__main__':
print(f"主进程 PID: {os.getpid()}")
print(f"CPU核心数: {os.cpu_count()}")
# 创建进程池
with ProcessPoolExecutor(max_workers=4) as executor:
# 提交单个任务
future1 = executor.submit(cpu_bound_task, 1000000)
print(f"任务已提交: {future1}")
# 提交多个任务
futures = [executor.submit(cpu_bound_task, n) for n in [500000, 1000000, 1500000, 2000000]]
# 获取结果
for i, future in enumerate(futures):
result = future.result() # 阻塞等待
print(f"任务 {i} 结果: {result}")
print("所有任务完成")
---
02.map方法并行处理
a.map方法特点
map方法按顺序返回结果，自动分配任务到进程池，适合批量处理同类任务。支持chunksize参数控制任务分块大小，提高效率。
b.map使用示例
---
from concurrent.futures import ProcessPoolExecutor
import time
import math
def is_prime(n):
if n < 2:
return False
for i in range(2, int(math.sqrt(n)) + 1):
if n % i == 0:
return False
return True
def find_primes_in_range(start, end):
pid = os.getpid()
primes = [n for n in range(start, end) if is_prime(n)]
print(f"进程 {pid} 找到 {len(primes)} 个质数 [{start}-{end})")
return primes
def process_data_chunk(chunk_id):
import os
pid = os.getpid()
print(f"进程 {pid} 处理数据块 {chunk_id}")
# 模拟数据处理
time.sleep(1)
result = sum(range(chunk_id * 1000, (chunk_id + 1) * 1000))
print(f"进程 {pid} 完成数据块 {chunk_id}, 结果: {result}")
return chunk_id, result
if __name__ == '__main__':
import os
# 示例1: 并行查找质数
print("=== 并行查找质数 ===")
ranges = [(1, 10000), (10000, 20000), (20000, 30000), (30000, 40000)]
start_time = time.time()
with ProcessPoolExecutor(max_workers=4) as executor:
results = executor.map(lambda r: find_primes_in_range(r[0], r[1]), ranges)
all_primes = []
for primes in results:
all_primes.extend(primes)
elapsed = time.time() - start_time
print(f"找到 {len(all_primes)} 个质数，耗时: {elapsed:.2f}秒")
# 示例2: 使用chunksize优化
print("\n=== 使用chunksize批量处理 ===")
chunk_ids = list(range(20))
start_time = time.time()
with ProcessPoolExecutor(max_workers=4) as executor:
# chunksize=5 表示每次分配5个任务给一个进程
results = executor.map(process_data_chunk, chunk_ids, chunksize=5)
for chunk_id, result in results:
print(f"数据块 {chunk_id}: {result}")
elapsed = time.time() - start_time
print(f"批量处理完成，耗时: {elapsed:.2f}秒")
---
03.进程池高级特性
a.异常处理与超时
Future对象的result方法支持timeout参数，超时抛出TimeoutError。任务异常会在result调用时重新抛出，需要捕获处理。
b.高级特性示例
---
from concurrent.futures import ProcessPoolExecutor, as_completed, wait, FIRST_COMPLETED
import time
import random
def task_with_exception(task_id):
import os
pid = os.getpid()
print(f"进程 {pid} 执行任务 {task_id}")
time.sleep(random.uniform(0.5, 2))
if task_id % 3 == 0:
raise ValueError(f"任务 {task_id} 故意失败")
return f"任务-{task_id}-成功"
def long_running_task(duration):
import os
pid = os.getpid()
print(f"进程 {pid} 开始长时间任务，预计 {duration} 秒")
time.sleep(duration)
return f"完成-{duration}秒任务"
if __name__ == '__main__':
# 示例1: 异常处理
print("=== 异常处理示例 ===")
with ProcessPoolExecutor(max_workers=3) as executor:
futures = [executor.submit(task_with_exception, i) for i in range(10)]
for i, future in enumerate(futures):
try:
result = future.result(timeout=5)
print(f"任务 {i} 成功: {result}")
except ValueError as e:
print(f"任务 {i} 失败: {e}")
except TimeoutError:
print(f"任务 {i} 超时")
# 示例2: as_completed按完成顺序处理
print("\n=== as_completed示例 ===")
with ProcessPoolExecutor(max_workers=3) as executor:
futures = {executor.submit(long_running_task, d): d for d in [3, 1, 2, 4, 1.5]}
for future in as_completed(futures):
duration = futures[future]
try:
result = future.result()
print(f"任务完成 (预计{duration}秒): {result}")
except Exception as e:
print(f"任务失败: {e}")
# 示例3: wait等待策略
print("\n=== wait等待策略示例 ===")
with ProcessPoolExecutor(max_workers=3) as executor:
futures = [executor.submit(long_running_task, d) for d in [2, 3, 1, 4]]
# 等待第一个完成
done, not_done = wait(futures, return_when=FIRST_COMPLETED)
print(f"第一个任务完成，已完成: {len(done)}, 未完成: {len(not_done)}")
# 等待所有完成
done, not_done = wait(not_done)
print(f"所有任务完成，已完成: {len(done)}, 未完成: {len(not_done)}")
# 获取所有结果
for future in futures:
print(f"结果: {future.result()}")
---
3. 协程与异步编程
3.1 asyncio事件循环
01.事件循环基础
a.事件循环概念
事件循环是asyncio的核心，负责调度协程执行、处理IO事件、管理回调函数。通过asyncio.run启动事件循环，自动管理循环生命周期。
b.基础使用示例
---
import asyncio
import time
async def hello():
print("Hello")
await asyncio.sleep(1)
print("World")
return "完成"
async def count_down(n):
for i in range(n, 0, -1):
print(f"倒计时: {i}")
await asyncio.sleep(1)
print("倒计时结束")
async def main():
print(f"开始时间: {time.strftime('%H:%M:%S')}")
# 运行单个协程
result = await hello()
print(f"结果: {result}")
# 运行另一个协程
await count_down(3)
print(f"结束时间: {time.strftime('%H:%M:%S')}")
# 启动事件循环
asyncio.run(main())
---
02.并发执行协程
a.gather和create_task
gather并发执行多个协程并收集结果，create_task创建任务对象立即调度执���。gather按顺序返回结果，create_task返回Task对象可单独管理。
b.并发执行示例
---
import asyncio
import time
async def fetch_data(source, delay):
print(f"开始获取 {source} 数据")
await asyncio.sleep(delay)
print(f"{source} 数据获取完成")
return f"{source}-数据"
async def process_data(data, delay):
print(f"开始处理 {data}")
await asyncio.sleep(delay)
print(f"{data} 处理完成")
return f"{data}-已处理"
async def main():
start = time.time()
# 使用gather并发执行
print("=== 使用gather并发执行 ===")
results = await asyncio.gather(
fetch_data("API-1", 2),
fetch_data("API-2", 1),
fetch_data("API-3", 3)
)
print(f"gather结果: {results}")
# 使用create_task
print("\n=== 使用create_task ===")
task1 = asyncio.create_task(fetch_data("DB-1", 1.5))
task2 = asyncio.create_task(fetch_data("DB-2", 1))
task3 = asyncio.create_task(fetch_data("DB-3", 2))
# 等待所有任务完成
result1 = await task1
result2 = await task2
result3 = await task3
print(f"task结果: {result1}, {result2}, {result3}")
# 链式处理
print("\n=== 链式处理 ===")
data = await fetch_data("Source", 1)
processed = await process_data(data, 1)
print(f"最终结果: {processed}")
elapsed = time.time() - start
print(f"\n总耗时: {elapsed:.2f}秒")
asyncio.run(main())
---
03.事件循环控制
a.循环管理方法
get_event_loop获取当前循环，new_event_loop创建新循环，set_event_loop设置当前循环。run_until_complete运行协程直到完成，stop停止循环，close关闭循环。
b.循环控制示例
---
import asyncio
import time
async def worker(name, duration):
print(f"{name} 开始工作")
await asyncio.sleep(duration)
print(f"{name} 工作完成")
return f"{name}-结果"
async def monitor():
for i in range(5):
print(f"[监控] 循环运行中 {i+1}/5")
await asyncio.sleep(1)
def callback(future):
print(f"回调函数: 任务完成，��果={future.result()}")
async def main():
# 获取当前事件循环
loop = asyncio.get_event_loop()
print(f"事件循环: {loop}")
print(f"循环运行中: {loop.is_running()}")
# 创建任务并添加回调
task = asyncio.create_task(worker("Worker-1", 2))
task.add_done_callback(callback)
# 并发运行多个任务
tasks = [
asyncio.create_task(worker(f"Worker-{i}", i))
for i in range(2, 5)
]
# 启动监控
monitor_task = asyncio.create_task(monitor())
# 等待所有任务
await task
results = await asyncio.gather(*tasks)
print(f"所有worker结果: {results}")
# 取消监控任务
monitor_task.cancel()
try:
await monitor_task
except asyncio.CancelledError:
print("监控任务已取消")
# 使用run启动
asyncio.run(main())
# 手动管理循环示例
print("\n=== 手动管理循环 ===")
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
result = loop.run_until_complete(worker("Manual", 1))
print(f"手动循环结果: {result}")
finally:
loop.close()
---
3.2 async与await语法
01.协程函数定义
a.async def语法
使用async def定义协程函数，返回协程对象。协程函数内部可使用await暂停执行，等待其他协程完成。协程必须在事件循环中运行。
b.协程定义示例
---
import asyncio
# 定义协程函数
async def greet(name, delay):
print(f"Hello {name}")
await asyncio.sleep(delay)
print(f"Goodbye {name}")
return f"{name} 完成"
# 协程组合
async def fetch_user(user_id):
print(f"获取用户 {user_id}")
await asyncio.sleep(1)
return {"id": user_id, "name": f"User-{user_id}"}
async def fetch_orders(user_id):
print(f"获取订单 {user_id}")
await asyncio.sleep(1.5)
return [f"Order-{i}" for i in range(3)]
async def get_user_data(user_id):
# 串行执行
user = await fetch_user(user_id)
orders = await fetch_orders(user_id)
return {"user": user, "orders": orders}
async def main():
# 调用协程
result = await greet("Alice", 1)
print(f"结果: {result}")
# 获取用户数据
data = await get_user_data(100)
print(f"用户数据: {data}")
asyncio.run(main())
---
02.await表达式
a.await用法
await只能在async函数内使用，用于等待可等待对象完成。可等待对象包括协程、Task、Future。await会暂停当前协程，将控制权交还事件循环。
b.await使用示例
---
import asyncio
import time
async def compute(x, y):
print(f"计算 {x} + {y}")
await asyncio.sleep(1)
return x + y
async def fetch_api(url):
print(f"请求 {url}")
await asyncio.sleep(2)
return f"{url} 响应数据"
async def main():
start = time.time()
# 串行执行 - 总耗时3秒
print("=== 串行执行 ===")
result1 = await compute(10, 20)
result2 = await fetch_api("https://api.example.com")
print(f"串行结果: {result1}, {result2}")
print(f"串行耗时: {time.time() - start:.2f}秒\n")
# 并行执行 - 总耗时2秒
print("=== 并行执行 ===")
start = time.time()
task1 = asyncio.create_task(compute(30, 40))
task2 = asyncio.create_task(fetch_api("https://api.example.com/data"))
result1 = await task1
result2 = await task2
print(f"并行结果: {result1}, {result2}")
print(f"并行耗时: {time.time() - start:.2f}秒")
asyncio.run(main())
---
03.异步上下文管理器
a.async with语法
async with用于异步上下文管理器，支持__aenter__和__aexit__异步方法。适用于异步资源管理，如数据库连接、文件操作。
b.异步上下文示例
---
import asyncio
class AsyncResource:
def __init__(self, name):
self.name = name
async def __aenter__(self):
print(f"{self.name} 资源获取中...")
await asyncio.sleep(1)
print(f"{self.name} 资源已获取")
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
print(f"{self.name} 资源释放中...")
await asyncio.sleep(0.5)
print(f"{self.name} 资源已释放")
async def use(self):
print(f"使用 {self.name} 资源")
await asyncio.sleep(1)
class AsyncDatabase:
def __init__(self, db_name):
self.db_name = db_name
self.connection = None
async def __aenter__(self):
print(f"连接数据库 {self.db_name}")
await asyncio.sleep(0.5)
self.connection = f"Connection-{self.db_name}"
print(f"数据库连接成功: {self.connection}")
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
print(f"关闭数据库连接 {self.connection}")
await asyncio.sleep(0.3)
self.connection = None
print("数据库连接已关闭")
async def query(self, sql):
print(f"执行查询: {sql}")
await asyncio.sleep(1)
return [{"id": 1, "data": "结果"}]
async def main():
# 使用异步上下文管理器
async with AsyncResource("FileHandler") as resource:
await resource.use()
print()
# 数据库操作示例
async with AsyncDatabase("users_db") as db:
results = await db.query("SELECT * FROM users")
print(f"查询结果: {results}")
print()
# 嵌套使用
async with AsyncResource("Cache") as cache:
async with AsyncDatabase("cache_db") as db:
await cache.use()
data = await db.query("SELECT * FROM cache")
print(f"缓存数据: {data}")
asyncio.run(main())
---
3.3 协程任务管理
01.Task任务创建
a.create_task方法
create_task将协程包装为Task对象并立即调度执行，返回Task对象可用于取消、查询状态。Task继承自Future，支持add_done_callback添加完成回调。
b.任务创建示例
---
import asyncio
import time
async def compute(name, x, y, delay):
print(f"{name} 开始计算 {x} + {y}")
await asyncio.sleep(delay)
result = x + y
print(f"{name} 计算完成: {result}")
return result
async def fetch_data(source, delay):
print(f"获取 {source} 数据")
await asyncio.sleep(delay)
print(f"{source} 数据获取完成")
return f"{source}-数据"
def task_callback(task):
try:
result = task.result()
print(f"回调: 任务完成，结果={result}")
except Exception as e:
print(f"回调: 任务失败，错误={e}")
async def main():
start = time.time()
# 创建任务
task1 = asyncio.create_task(compute("Task-1", 10, 20, 1))
task2 = asyncio.create_task(compute("Task-2", 30, 40, 2))
task3 = asyncio.create_task(fetch_data("API", 1.5))
# 添加回调
task1.add_done_callback(task_callback)
task2.add_done_callback(task_callback)
# 查询任务状态
print(f"Task-1 状态: done={task1.done()}, cancelled={task1.cancelled()}")
# 等待任务完成
result1 = await task1
result2 = await task2
result3 = await task3
print(f"\n结果: {result1}, {result2}, {result3}")
print(f"总耗时: {time.time() - start:.2f}秒")
# 批量创建任务
tasks = [
asyncio.create_task(compute(f"Batch-{i}", i, i*2, 0.5))
for i in range(5)
]
# 等待所有任务
results = await asyncio.gather(*tasks)
print(f"批量结果: {results}")
asyncio.run(main())
---
02.任务取消与超时
a.任务取消机制
Task.cancel取消任务，任务内await处抛出CancelledError异常。可通过try-except捕获清理资源。asyncio.wait_for设置超时，超时抛出TimeoutError。
b.取消超时示例
---
import asyncio
import time
async def long_task(name, duration):
try:
print(f"{name} 开始，预计 {duration} 秒")
for i in range(duration):
print(f"{name} 进度: {i+1}/{duration}")
await asyncio.sleep(1)
print(f"{name} 正常完成")
return f"{name}-完成"
except asyncio.CancelledError:
print(f"{name} 被取消，执行清理")
await asyncio.sleep(0.5) # 清理操作
print(f"{name} 清理完成")
raise
async def cancellable_task(task_id):
try:
print(f"任务 {task_id} 启动")
await asyncio.sleep(5)
return f"任务-{task_id}-完成"
except asyncio.CancelledError:
print(f"任务 {task_id} 取消")
raise
async def main():
# 示例1: 手动取消任务
print("=== 手动取消示例 ===")
task = asyncio.create_task(long_task("LongTask", 10))
await asyncio.sleep(3)
print("取消任务...")
task.cancel()
try:
await task
except asyncio.CancelledError:
print("任务已取消")
# 示例2: 超时控制
print("\n=== 超时控制示例 ===")
try:
result = await asyncio.wait_for(
long_task("TimeoutTask", 10),
timeout=2.5
)
print(f"结果: {result}")
except asyncio.TimeoutError:
print("任务超时")
# 示例3: 批量取消
print("\n=== 批量取消示例 ===")
tasks = [
asyncio.create_task(cancellable_task(i))
for i in range(5)
]
await asyncio.sleep(1)
print("取消所有任务...")
for task in tasks:
task.cancel()
# 等待所有任务完成取消
results = await asyncio.gather(*tasks, return_exceptions=True)
print(f"取消结果: {results}")
# 示例4: shield保护任务
print("\n=== shield保护示例 ===")
async def protected_task():
print("受保护任务开始")
await asyncio.sleep(2)
print("受保护任务完成")
return "protected-result"
task = asyncio.create_task(protected_task())
shielded = asyncio.shield(task)
try:
await asyncio.wait_for(shielded, timeout=1)
except asyncio.TimeoutError:
print("等待超时，但任务继续运行")
result = await task
print(f"最终结果: {result}")
asyncio.run(main())
---
03.TaskGroup任务组
a.TaskGroup特性
Python3.11引入TaskGroup，提供结构化并发管理。自动等待所有任务完成，任一任务异常则取消其他任务。使用async with管理生命周期。
b.TaskGroup使用示例
---
import asyncio
import random
async def worker(name, duration, fail=False):
print(f"{name} 开始，耗时 {duration} 秒")
await asyncio.sleep(duration)
if fail:
raise ValueError(f"{name} 故意失败")
print(f"{name} 完成")
return f"{name}-结果"
async def fetch_resource(resource_id, delay):
print(f"获取资源 {resource_id}")
await asyncio.sleep(delay)
if random.random() < 0.2:
raise ConnectionError(f"资源 {resource_id} 连接失败")
return {"id": resource_id, "data": f"数据-{resource_id}"}
async def main():
# 示例1: 基本TaskGroup使用
print("=== 基本TaskGroup ===")
try:
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(worker("Worker-1", 1))
task2 = tg.create_task(worker("Worker-2", 2))
task3 = tg.create_task(worker("Worker-3", 1.5))
# 所有任务完成后才会到这里
print(f"所有任务完成")
print(f"结果: {task1.result()}, {task2.result()}, {task3.result()}")
except Exception as e:
print(f"TaskGroup 异常: {e}")
# 示例2: 异常处理
print("\n=== 异常处理示例 ===")
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(worker("Normal-1", 1))
tg.create_task(worker("Fail", 0.5, fail=True))
tg.create_task(worker("Normal-2", 2))
except* ValueError as eg:
print(f"捕获 ValueError 异常组: {len(eg.exceptions)} 个")
for e in eg.exceptions:
print(f" - {e}")
# 示例3: 动态添加任务
print("\n=== 动态添加任务 ===")
async with asyncio.TaskGroup() as tg:
# 初始任务
for i in range(3):
tg.create_task(worker(f"Initial-{i}", random.uniform(0.5, 1.5)))
await asyncio.sleep(0.5)
# 动态添加更多任务
for i in range(3, 6):
tg.create_task(worker(f"Dynamic-{i}", random.uniform(0.5, 1.5)))
print("所有动态任务完成")
# 示例4: 资源获取场景
print("\n=== 资源获取场景 ===")
resource_ids = list(range(10))
results = []
try:
async with asyncio.TaskGroup() as tg:
tasks = [
tg.create_task(fetch_resource(rid, random.uniform(0.5, 2)))
for rid in resource_ids
]
# 收集结果
for task in tasks:
try:
results.append(task.result())
except Exception as e:
print(f"任务失败: {e}")
except* ConnectionError as eg:
print(f"连接错误: {len(eg.exceptions)} 个资源失败")
print(f"成功获取 {len(results)} 个资源")
# Python 3.11+ 支持
if hasattr(asyncio, 'TaskGroup'):
asyncio.run(main())
else:
print("TaskGroup 需要 Python 3.11+")
---
3.4 异步IO操作
01.异步文件操作
a.aiofiles库
aiofiles提供异步文件读写接口，避免阻塞事件循环。支持open、read、write、readline等异步方法，使用async with管理文件生命周期。
b.异步文件示例
---
import asyncio
import aiofiles
import time
async def write_file(filename, content):
print(f"写入文件: {filename}")
async with aiofiles.open(filename, 'w', encoding='utf-8') as f:
await f.write(content)
print(f"写入完成: {filename}")
async def read_file(filename):
print(f"读取文件: {filename}")
async with aiofiles.open(filename, 'r', encoding='utf-8') as f:
content = await f.read()
print(f"读取完成: {filename}, 长度: {len(content)}")
return content
async def append_file(filename, content):
async with aiofiles.open(filename, 'a', encoding='utf-8') as f:
await f.write(content)
async def read_lines(filename):
lines = []
async with aiofiles.open(filename, 'r', encoding='utf-8') as f:
async for line in f:
lines.append(line.strip())
return lines
async def main():
start = time.time()
# 并发写入多个文件
write_tasks = [
write_file(f"file_{i}.txt", f"内容 {i}\n" * 100)
for i in range(5)
]
await asyncio.gather(*write_tasks)
# 并发读取文件
read_tasks = [
read_file(f"file_{i}.txt")
for i in range(5)
]
contents = await asyncio.gather(*read_tasks)
# 追加内容
await append_file("file_0.txt", "\n追加内容")
# 按行读取
lines = await read_lines("file_0.txt")
print(f"文件行数: {len(lines)}")
elapsed = time.time() - start
print(f"总耗时: {elapsed:.2f}秒")
asyncio.run(main())
---
02.异步网络请求
a.aiohttp库
aiohttp提供异步HTTP客户端和服务器。ClientSession管理连接池，支持get、post等方法。使用async with管理会话和响应生命周期。
b.异步请求示例
---
import asyncio
import aiohttp
import time
async def fetch_url(session, url):
print(f"请求: {url}")
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
status = response.status
text = await response.text()
print(f"完成: {url}, 状态: {status}, 长度: {len(text)}")
return {"url": url, "status": status, "length": len(text)}
except asyncio.TimeoutError:
print(f"超时: {url}")
return {"url": url, "error": "timeout"}
except Exception as e:
print(f"错误: {url}, {e}")
return {"url": url, "error": str(e)}
async def post_data(session, url, data):
print(f"POST: {url}")
async with session.post(url, json=data) as response:
result = await response.json()
print(f"POST完成: {url}")
return result
async def fetch_with_headers(session, url, headers):
async with session.get(url, headers=headers) as response:
return await response.text()
async def main():
start = time.time()
# 创建会话
async with aiohttp.ClientSession() as session:
# 并发请求多个URL
urls = [
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2",
"https://httpbin.org/delay/1",
"https://httpbin.org/status/200",
"https://httpbin.org/status/404"
]
tasks = [fetch_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
print(f"\n请求结果:")
for result in results:
print(f" {result}")
# POST请求
post_result = await post_data(
session,
"https://httpbin.org/post",
{"key": "value", "data": [1, 2, 3]}
)
print(f"\nPOST结果: {post_result}")
# 自定义请求头
headers = {"User-Agent": "AsyncClient/1.0"}
content = await fetch_with_headers(
session,
"https://httpbin.org/headers",
headers
)
print(f"\n自定义请求头响应长度: {len(content)}")
elapsed = time.time() - start
print(f"\n总耗时: {elapsed:.2f}秒")
asyncio.run(main())
---
03.异步数据库操作
a.asyncpg库
asyncpg提供PostgreSQL异步驱动，性能优于psycopg2。支持连接池、预处理语句、事务管理。使用async with管理连接和事务。
b.异步数据库示例
---
import asyncio
import asyncpg
async def create_table(conn):
await conn.execute('''
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
email VARCHAR(100),
age INTEGER
)
''')
print("表创建完成")
async def insert_user(conn, name, email, age):
await conn.execute(
'INSERT INTO users(name, email, age) VALUES($1, $2, $3)',
name, email, age
)
print(f"插入用户: {name}")
async def batch_insert(conn, users):
await conn.executemany(
'INSERT INTO users(name, email, age) VALUES($1, $2, $3)',
users
)
print(f"批量插入 {len(users)} 个用户")
async def query_users(conn):
rows = await conn.fetch('SELECT * FROM users')
print(f"查询到 {len(rows)} 个用户:")
for row in rows:
print(f" {dict(row)}")
return rows
async def query_user_by_id(conn, user_id):
row = await conn.fetchrow('SELECT * FROM users WHERE id = $1', user_id)
if row:
print(f"用户 {user_id}: {dict(row)}")
return row
async def update_user(conn, user_id, name):
await conn.execute(
'UPDATE users SET name = $1 WHERE id = $2',
name, user_id
)
print(f"更新用户 {user_id}")
async def transaction_example(conn):
async with conn.transaction():
await conn.execute(
'INSERT INTO users(name, email, age) VALUES($1, $2, $3)',
'Transaction User', 'trans@example.com', 30
)
await conn.execute(
'UPDATE users SET age = age + 1 WHERE name = $1',
'Transaction User'
)
print("事务提交成功")
async def main():
# 连接数据库
conn = await asyncpg.connect(
host='localhost',
port=5432,
user='postgres',
password='password',
database='testdb'
)
try:
# 创建表
await create_table(conn)
# 插入单个用户
await insert_user(conn, 'Alice', 'alice@example.com', 25)
await insert_user(conn, 'Bob', 'bob@example.com', 30)
# 批量插入
users = [
('Charlie', 'charlie@example.com', 28),
('David', 'david@example.com', 35),
('Eve', 'eve@example.com', 22)
]
await batch_insert(conn, users)
# 查询所有用户
await query_users(conn)
# 查询单个用户
await query_user_by_id(conn, 1)
# 更新用户
await update_user(conn, 1, 'Alice Updated')
# 事务示例
await transaction_example(conn)
# 最终查询
await query_users(conn)
finally:
await conn.close()
print("数据库连接已关闭")
# 使用连接池
async def pool_example():
pool = await asyncpg.create_pool(
host='localhost',
port=5432,
user='postgres',
password='password',
database='testdb',
min_size=5,
max_size=20
)
try:
# 并发查询
async def query_task(pool, user_id):
async with pool.acquire() as conn:
return await query_user_by_id(conn, user_id)
tasks = [query_task(pool, i) for i in range(1, 6)]
results = await asyncio.gather(*tasks)
print(f"并发查询完成: {len(results)} 个结果")
finally:
await pool.close()
print("连接池已关闭")
# asyncio.run(main())
# asyncio.run(pool_example())
---
3.5 异步迭代器与生成器
01.异步迭代器
a.异步迭代协议
异步迭代器实现__aiter__返回自身，__anext__返回awaitable对象。使用async for遍历异步迭代器，StopAsyncIteration标记迭代结束。
b.异步迭代器示例
---
import asyncio
class AsyncRange:
def __init__(self, start, end, delay=0.1):
self.current = start
self.end = end
self.delay = delay
def __aiter__(self):
return self
async def __anext__(self):
if self.current >= self.end:
raise StopAsyncIteration
await asyncio.sleep(self.delay)
value = self.current
self.current += 1
return value
class AsyncDataStream:
def __init__(self, data_source, chunk_size=10):
self.data_source = data_source
self.chunk_size = chunk_size
self.index = 0
def __aiter__(self):
return self
async def __anext__(self):
if self.index >= len(self.data_source):
raise StopAsyncIteration
# 模拟异步数据获取
await asyncio.sleep(0.1)
end = min(self.index + self.chunk_size, len(self.data_source))
chunk = self.data_source[self.index:end]
self.index = end
return chunk
async def main():
# 使用AsyncRange
print("=== AsyncRange示例 ===")
async for i in AsyncRange(0, 5, 0.2):
print(f"值: {i}")
# 使用AsyncDataStream
print("\n=== AsyncDataStream示例 ===")
data = list(range(50))
stream = AsyncDataStream(data, chunk_size=10)
async for chunk in stream:
print(f"数据块: {chunk}")
# 手动迭代
print("\n=== 手动迭代示例 ===")
iterator = AsyncRange(10, 15).__aiter__()
while True:
try:
value = await iterator.__anext__()
print(f"手动获取: {value}")
except StopAsyncIteration:
print("迭代结束")
break
asyncio.run(main())
---
02.异步生成器
a.async def与yield
使用async def和yield定义异步生成器，自动实现异步迭代协议。支持yield表达式暂停执行，await等待异步操作。比手动实现迭代器更简洁。
b.异步生成器示例
---
import asyncio
import random
async def async_range(start, end, delay=0.1):
for i in range(start, end):
await asyncio.sleep(delay)
yield i
async def fetch_pages(url, max_pages):
for page in range(1, max_pages + 1):
print(f"获取页面 {page}")
await asyncio.sleep(0.5)
yield {
"url": f"{url}?page={page}",
"data": [f"item-{i}" for i in range(10)]
}
async def monitor_system(duration, interval=1):
import time
start = time.time()
while time.time() - start < duration:
await asyncio.sleep(interval)
yield {
"timestamp": time.time(),
"cpu": random.uniform(0, 100),
"memory": random.uniform(0, 100)
}
async def async_filter(async_iter, predicate):
async for item in async_iter:
if predicate(item):
yield item
async def async_map(async_iter, func):
async for item in async_iter:
yield func(item)
async def main():
# 基本异步生成器
print("=== 基本异步生成器 ===")
async for i in async_range(0, 5, 0.2):
print(f"生成: {i}")
# 分页数据获取
print("\n=== 分页数据获取 ===")
async for page in fetch_pages("https://api.example.com/data", 3):
print(f"页面: {page['url']}, 项数: {len(page['data'])}")
# 系统监控
print("\n=== 系统监控 ===")
async for metrics in monitor_system(3, 0.5):
print(f"时间: {metrics['timestamp']:.2f}, "
f"CPU: {metrics['cpu']:.1f}%, "
f"内存: {metrics['memory']:.1f}%")
# 异步生成器组合
print("\n=== 异步生成器组合 ===")
numbers = async_range(0, 20, 0.1)
evens = async_filter(numbers, lambda x: x % 2 == 0)
squared = async_map(evens, lambda x: x ** 2)
async for value in squared:
print(f"偶数平方: {value}")
asyncio.run(main())
---
03.异步推导式
a.异步推导语法
支持async for的列表、集合、字典推导式。使用[x async for x in ...]语法，需在async函数内使用。可结合if条件过滤，支持嵌套推导。
b.异步推导示例
---
import asyncio
async def async_range(n):
for i in range(n):
await asyncio.sleep(0.1)
yield i
async def fetch_data(item_id):
await asyncio.sleep(0.1)
return {"id": item_id, "value": item_id * 10}
async def async_items():
items = ["a", "b", "c", "d", "e"]
for item in items:
await asyncio.sleep(0.1)
yield item
async def main():
# 异步列表推导
print("=== 异步列表推导 ===")
numbers = [x async for x in async_range(10)]
print(f"数字列表: {numbers}")
# 带条件的异步推导
evens = [x async for x in async_range(20) if x % 2 == 0]
print(f"偶数列表: {evens}")
# 异步集合推导
print("\n=== 异步集合推导 ===")
unique_values = {x % 5 async for x in async_range(15)}
print(f"唯一值集合: {unique_values}")
# 异步字典推导
print("\n=== 异步字典推导 ===")
data_dict = {
x: x ** 2
async for x in async_range(5)
}
print(f"字典: {data_dict}")
# 嵌套异步推导
print("\n=== 嵌套异步推导 ===")
matrix = [
[i * j async for j in async_range(3)]
async for i in async_range(3)
]
print(f"矩阵: {matrix}")
# 异步推导与await结合
print("\n=== 异步推导与await ===")
results = [
await fetch_data(i)
async for i in async_range(5)
]
print(f"获取结果: {results}")
# 字符串处理
print("\n=== 字符串处理 ===")
upper_items = [
item.upper()
async for item in async_items()
]
print(f"大写列表: {upper_items}")
# 过滤和转换
print("\n=== 过滤和转换 ===")
filtered = [
x * 2
async for x in async_range(20)
if x % 3 == 0
]
print(f"过滤转换结果: {filtered}")
asyncio.run(main())
---
4. concurrent.futures
4.1 Executor抽象类
01.Executor基础
a.Executor接口
Executor是concurrent.futures的抽象基类，定义submit和map方法。ThreadPoolExecutor和ProcessPoolExecutor继承Executor，提供统一的并发接口。
b.Executor使用示例
---
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import time
import os
def task(name, duration):
pid = os.getpid()
print(f"任务 {name} 开始, PID: {pid}")
time.sleep(duration)
result = f"{name}-完成-PID-{pid}"
print(f"任务 {name} 结束")
return result
def cpu_task(n):
pid = os.getpid()
print(f"CPU任务 {n} 开始, PID: {pid}")
result = sum(i * i for i in range(n))
print(f"CPU任务 {n} 完成, 结果: {result}")
return result
# 线程池Executor
print("=== ThreadPoolExecutor ===")
with ThreadPoolExecutor(max_workers=3) as executor:
# submit提交任务
future1 = executor.submit(task, "Task-1", 1)
future2 = executor.submit(task, "Task-2", 2)
future3 = executor.submit(task, "Task-3", 1.5)
# 获取结果
print(f"结果1: {future1.result()}")
print(f"结果2: {future2.result()}")
print(f"结果3: {future3.result()}")
# 进程池Executor
print("\n=== ProcessPoolExecutor ===")
if __name__ == '__main__':
with ProcessPoolExecutor(max_workers=3) as executor:
futures = [
executor.submit(cpu_task, n)
for n in [1000000, 2000000, 1500000]
]
for i, future in enumerate(futures):
result = future.result()
print(f"进程任务 {i} 结果: {result}")
---
02.submit方法
a.submit特性
submit提交单个任务到执行器，立即返回Future对象。Future代表异步计算结果，支持result获取结果、done查询状态、cancel取消任务。
b.submit使用示例
---
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
import random
def download_file(file_id, size):
print(f"下载文件 {file_id}, 大小: {size}MB")
time.sleep(random.uniform(1, 3))
return {"file_id": file_id, "size": size, "status": "success"}
def process_data(data_id):
print(f"处理数据 {data_id}")
time.sleep(random.uniform(0.5, 2))
if random.random() < 0.2:
raise ValueError(f"数据 {data_id} 处理失败")
return f"数据-{data_id}-已处理"
with ThreadPoolExecutor(max_workers=5) as executor:
# 提交多个任务
futures = {}
for i in range(10):
future = executor.submit(download_file, f"file-{i}", random.randint(10, 100))
futures[future] = f"file-{i}"
# 按完成顺序处理
print("\n=== 按完成顺序处理 ===")
for future in as_completed(futures):
file_id = futures[future]
try:
result = future.result()
print(f"{file_id} 完成: {result}")
except Exception as e:
print(f"{file_id} 失败: {e}")
# 处理数据任务
print("\n=== 数据处理任务 ===")
data_futures = [
executor.submit(process_data, i)
for i in range(5)
]
for i, future in enumerate(data_futures):
try:
result = future.result(timeout=5)
print(f"任务 {i}: {result}")
except ValueError as e:
print(f"任务 {i} 错误: {e}")
except TimeoutError:
print(f"任务 {i} 超时")
---
03.上下文管理器
a.with语句
Executor支持上下文管理器，使用with自动管理资源。退出时自动调用shutdown等待所有任务完成并清理资源。
b.上下文管理示例
---
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import time
def worker(task_id, duration):
print(f"Worker {task_id} 开始")
time.sleep(duration)
print(f"Worker {task_id} 完成")
return f"结果-{task_id}"
# 自动资源管理
print("=== 使用with自动管理 ===")
with ThreadPoolExecutor(max_workers=3) as executor:
futures = [
executor.submit(worker, i, 1)
for i in range(5)
]
results = [f.result() for f in futures]
print(f"所有结果: {results}")
# 退出with时自动shutdown
print("\n=== 手动管理 ===")
executor = ThreadPoolExecutor(max_workers=3)
try:
futures = [
executor.submit(worker, i, 0.5)
for i in range(3)
]
for future in futures:
print(future.result())
finally:
# 手动关闭
executor.shutdown(wait=True)
print("Executor已关闭")
# 嵌套使用
print("\n=== 嵌套Executor ===")
with ThreadPoolExecutor(max_workers=2) as outer_executor:
def outer_task(task_id):
print(f"外层任务 {task_id} 开始")
with ThreadPoolExecutor(max_workers=2) as inner_executor:
inner_futures = [
inner_executor.submit(worker, f"{task_id}-{i}", 0.5)
for i in range(2)
]
results = [f.result() for f in inner_futures]
print(f"外层任务 {task_id} 完成: {results}")
return results
outer_futures = [
outer_executor.submit(outer_task, i)
for i in range(2)
]
for future in outer_futures:
print(f"最终结果: {future.result()}")
---
4.2 Future对象
01.Future状态管理
a.Future状态方法
Future对象表示异步计算结果，提供done查询完成状态、running查询运行状态、cancelled查询取消状态。result获取结果会阻塞直到完成。
b.Future状态示例
---
from concurrent.futures import ThreadPoolExecutor
import time
def slow_task(task_id, duration):
print(f"任务 {task_id} 开始，耗时 {duration} 秒")
time.sleep(duration)
print(f"任务 {task_id} 完成")
return f"结果-{task_id}"
with ThreadPoolExecutor(max_workers=3) as executor:
# 提交任务
future1 = executor.submit(slow_task, 1, 2)
future2 = executor.submit(slow_task, 2, 1)
future3 = executor.submit(slow_task, 3, 3)
# 立即检查状态
print(f"Future1 - done: {future1.done()}, running: {future1.running()}")
print(f"Future2 - done: {future2.done()}, running: {future2.running()}")
time.sleep(0.5)
# 再次检查
print(f"\n0.5秒后:")
print(f"Future1 - done: {future1.done()}, running: {future1.running()}")
print(f"Future2 - done: {future2.done()}, running: {future2.running()}")
# 等待Future2完成
result2 = future2.result()
print(f"\nFuture2 结果: {result2}")
print(f"Future2 - done: {future2.done()}, running: {future2.running()}")
# 等待所有任务
result1 = future1.result()
result3 = future3.result()
print(f"\n所有结果: {result1}, {result2}, {result3}")
---
02.Future回调函数
a.add_done_callback方法
add_done_callback添加完成回调函数，任务完成时自动调用。回调函数接收Future对象作为参数，可获取结果或异常。支持添加多个回调。
b.回调函数示例
---
from concurrent.futures import ThreadPoolExecutor
import time
def task(task_id, duration):
print(f"任务 {task_id} 执行中")
time.sleep(duration)
if task_id == 3:
raise ValueError(f"任务 {task_id} 失败")
return f"任务-{task_id}-完成"
def success_callback(future):
try:
result = future.result()
print(f"[成功回调] 结果: {result}")
except Exception as e:
print(f"[成功回调] 异常: {e}")
def log_callback(future):
if future.done():
print(f"[日志回调] 任务已完成")
if future.cancelled():
print(f"[日志回调] 任务已取消")
def cleanup_callback(future):
print(f"[清理回调] 执行清理操作")
with ThreadPoolExecutor(max_workers=3) as executor:
# 任务1: 添加单个回调
future1 = executor.submit(task, 1, 1)
future1.add_done_callback(success_callback)
# 任务2: 添加多个回调
future2 = executor.submit(task, 2, 1.5)
future2.add_done_callback(success_callback)
future2.add_done_callback(log_callback)
future2.add_done_callback(cleanup_callback)
# 任务3: 失败任务的回调
future3 = executor.submit(task, 3, 0.5)
future3.add_done_callback(success_callback)
future3.add_done_callback(log_callback)
# 等待所有任务
time.sleep(2)
print("\n=== 链式回调示例 ===")
def chain_callback_1(future):
print(f"[回调1] 处理结果")
result = future.result()
print(f"[回调1] 结果: {result}")
def chain_callback_2(future):
print(f"[回调2] 后续处理")
def chain_callback_3(future):
print(f"[回调3] 最终清理")
future4 = executor.submit(task, 4, 1)
future4.add_done_callback(chain_callback_1)
future4.add_done_callback(chain_callback_2)
future4.add_done_callback(chain_callback_3)
future4.result()
---
03.Future取消与超时
a.cancel和result超时
cancel尝试取消任务，仅未开始执行的任务可取消。result支持timeout参数，超时抛出TimeoutError。exception获取任务异常。
b.取消超时示例
---
from concurrent.futures import ThreadPoolExecutor, TimeoutError
import time
def long_task(task_id, duration):
print(f"任务 {task_id} 开始，耗时 {duration} 秒")
for i in range(duration):
time.sleep(1)
print(f"任务 {task_id} 进度: {i+1}/{duration}")
return f"任务-{task_id}-完成"
with ThreadPoolExecutor(max_workers=2) as executor:
# 示例1: 取消未开始的任务
print("=== 取消任务示例 ===")
futures = [
executor.submit(long_task, i, 3)
for i in range(5)
]
time.sleep(0.1)
# 尝试取消后面的任务
for i, future in enumerate(futures[2:], start=2):
cancelled = future.cancel()
print(f"任务 {i} 取消: {cancelled}")
# 等待未取消的任务
for i, future in enumerate(futures[:2]):
try:
result = future.result()
print(f"任务 {i} 结果: {result}")
except Exception as e:
print(f"任务 {i} 异常: {e}")
# 示例2: 超时处理
print("\n=== 超时处理示例 ===")
future = executor.submit(long_task, 10, 5)
try:
result = future.result(timeout=2)
print(f"结果: {result}")
except TimeoutError:
print("任务超时")
print(f"任务状态 - done: {future.done()}, running: {future.running()}")
# 示例3: 异常处理
print("\n=== 异常处理示例 ===")
def error_task(task_id):
print(f"任务 {task_id} 开始")
time.sleep(1)
raise RuntimeError(f"任务 {task_id} 发生错误")
future = executor.submit(error_task, 20)
try:
result = future.result()
except RuntimeError as e:
print(f"捕获异常: {e}")
# 使用exception方法
exc = future.exception()
if exc:
print(f"Future异常: {exc}")
# 示例4: 批量超时处理
print("\n=== 批量超时处理 ===")
futures = [
executor.submit(long_task, i, i+1)
for i in range(3)
]
for i, future in enumerate(futures):
try:
result = future.result(timeout=2)
print(f"任务 {i} 成功: {result}")
except TimeoutError:
print(f"任务 {i} 超时")
except Exception as e:
print(f"任务 {i} 异常: {e}")
---
4.3 as_completed与wait
01.as_completed迭代器
a.as_completed特性
as_completed返回迭代器，按Future完成顺序yield结果。适合需要立即处理完成任务的场景，不按提交顺序返回。支持timeout参数设置超时。
b.as_completed示例
---
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
import random
def download_task(file_id, size):
duration = random.uniform(1, 4)
print(f"下载 {file_id}, 大小: {size}MB, 预计: {duration:.1f}秒")
time.sleep(duration)
return {"file_id": file_id, "size": size, "duration": duration}
with ThreadPoolExecutor(max_workers=5) as executor:
# 提交任务
futures = {
executor.submit(download_task, f"file-{i}", random.randint(10, 100)): i
for i in range(10)
}
print("=== 按完成顺序处理 ===")
for future in as_completed(futures):
task_id = futures[future]
try:
result = future.result()
print(f"任务 {task_id} 完成: {result}")
except Exception as e:
print(f"任务 {task_id} 失败: {e}")
# 带超时的as_completed
print("\n=== 带超时的as_completed ===")
futures = [
executor.submit(download_task, f"file-{i}", 50)
for i in range(5)
]
try:
for future in as_completed(futures, timeout=3):
result = future.result()
print(f"完成: {result['file_id']}")
except TimeoutError:
print("部分任务超时")
# 实时进度显示
print("\n=== 实时进度显示 ===")
total = 20
futures = [
executor.submit(download_task, f"file-{i}", random.randint(10, 50))
for i in range(total)
]
completed = 0
for future in as_completed(futures):
completed += 1
result = future.result()
progress = (completed / total) * 100
print(f"进度: {progress:.1f}% ({completed}/{total}) - {result['file_id']}")
---
02.wait函数
a.wait等待策略
wait等待Future完成，返回done和not_done集合。支持FIRST_COMPLETED、FIRST_EXCEPTION、ALL_COMPLETED策略。timeout参数设置等待超时。
b.wait使用示例
---
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED, FIRST_EXCEPTION, ALL_COMPLETED
import time
import random
def task(task_id, duration, fail=False):
print(f"任务 {task_id} 开始")
time.sleep(duration)
if fail:
raise ValueError(f"任务 {task_id} 失败")
print(f"任务 {task_id} 完成")
return f"结果-{task_id}"
with ThreadPoolExecutor(max_workers=5) as executor:
# 示例1: FIRST_COMPLETED
print("=== FIRST_COMPLETED ===")
futures = [
executor.submit(task, i, random.uniform(1, 3))
for i in range(5)
]
done, not_done = wait(futures, return_when=FIRST_COMPLETED)
print(f"第一个完成，已完成: {len(done)}, 未完成: {len(not_done)}")
for future in done:
print(f"结果: {future.result()}")
# 等待剩余任务
done, not_done = wait(not_done)
print(f"全部完成，已完成: {len(done)}, 未完成: {len(not_done)}")
# 示例2: FIRST_EXCEPTION
print("\n=== FIRST_EXCEPTION ===")
futures = [
executor.submit(task, i, 1, fail=(i == 2))
for i in range(5)
]
done, not_done = wait(futures, return_when=FIRST_EXCEPTION)
print(f"首个异常，已完成: {len(done)}, 未完成: {len(not_done)}")
for future in done:
try:
result = future.result()
print(f"成功: {result}")
except Exception as e:
print(f"异常: {e}")
# 示例3: ALL_COMPLETED with timeout
print("\n=== ALL_COMPLETED with timeout ===")
futures = [
executor.submit(task, i, i+1)
for i in range(5)
]
done, not_done = wait(futures, timeout=3, return_when=ALL_COMPLETED)
print(f"3秒后，已完成: {len(done)}, 未完成: {len(not_done)}")
# 处理已完成的
for future in done:
print(f"已完成: {future.result()}")
# 等待剩余的
if not_done:
done, not_done = wait(not_done)
for future in done:
print(f"后续完成: {future.result()}")
---
03.as_completed与wait对比
a.使用场景对比
as_completed适合需要立即处理每个完成任务的场景，按完成顺序迭代。wait适合批量等待、条件等待场景，返回集合便于分组处理。
b.对比示例
---
from concurrent.futures import ThreadPoolExecutor, as_completed, wait, FIRST_COMPLETED
import time
import random
def process_task(task_id):
duration = random.uniform(0.5, 2)
print(f"任务 {task_id} 处理中，耗时 {duration:.1f}秒")
time.sleep(duration)
return {"task_id": task_id, "result": task_id * 10}
with ThreadPoolExecutor(max_workers=4) as executor:
# 场景1: 使用as_completed实时处理
print("=== as_completed: 实时处理 ===")
start = time.time()
futures = [executor.submit(process_task, i) for i in range(10)]
results = []
for future in as_completed(futures):
result = future.result()
results.append(result)
print(f"实时处理: {result}, 已完成: {len(results)}/10")
elapsed = time.time() - start
print(f"as_completed 总耗时: {elapsed:.2f}秒\n")
# 场景2: 使用wait批量处理
print("=== wait: 批量处理 ===")
start = time.time()
futures = [executor.submit(process_task, i) for i in range(10)]
# 等待前5个完成
done, not_done = wait(futures, return_when=FIRST_COMPLETED)
while len(done) < 5 and not_done:
new_done, not_done = wait(not_done, return_when=FIRST_COMPLETED)
done.update(new_done)
print(f"前5个任务完成，开始批量处理")
batch1_results = [f.result() for f in done]
print(f"批次1结果: {batch1_results}")
# 等待剩余任务
done, not_done = wait(not_done)
batch2_results = [f.result() for f in done]
print(f"批次2结果: {batch2_results}")
elapsed = time.time() - start
print(f"wait 总耗时: {elapsed:.2f}秒\n")
# 场景3: 混合使用
print("=== 混合使用 ===")
futures = [executor.submit(process_task, i) for i in range(8)]
# 先等待至少3个完成
done, not_done = wait(futures, return_when=FIRST_COMPLETED)
while len(done) < 3:
new_done, not_done = wait(not_done, return_when=FIRST_COMPLETED)
done.update(new_done)
print(f"至少3个完成，开始处理")
# 对已完成的立即处理
for future in done:
result = future.result()
print(f"早期完成: {result}")
# 对剩余的按完成顺序处理
for future in as_completed(not_done):
result = future.result()
print(f"后续完成: {result}")
---
4.4 map方法批量处理
01.map方法基础
a.map特性
map方法批量提交任务，按输入顺序返回结果。类似内置map但支持并发执行，返回迭代器延迟获取结果。适合批量处理同类任务。
b.map基础示例
---
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import time
def square(n):
print(f"计算 {n} 的平方")
time.sleep(0.5)
return n * n
def process_url(url):
print(f"处理 {url}")
time.sleep(1)
return {"url": url, "status": 200, "length": len(url) * 100}
# 线程池map
print("=== ThreadPoolExecutor.map ===")
with ThreadPoolExecutor(max_workers=4) as executor:
numbers = range(1, 11)
results = executor.map(square, numbers)
# 按顺序获取结果
for num, result in zip(numbers, results):
print(f"{num}² = {result}")
# 进程池map
print("\n=== ProcessPoolExecutor.map ===")
if __name__ == '__main__':
with ProcessPoolExecutor(max_workers=4) as executor:
numbers = range(1, 6)
results = list(executor.map(square, numbers))
print(f"结果: {results}")
# 多参数map
print("\n=== 多参数map ===")
def power(base, exp):
result = base ** exp
print(f"{base}^{exp} = {result}")
return result
with ThreadPoolExecutor(max_workers=3) as executor:
bases = [2, 3, 4, 5]
exponents = [2, 3, 2, 4]
results = list(executor.map(power, bases, exponents))
print(f"幂运算结果: {results}")
---
02.map超时与异常
a.timeout参数
map支持timeout参数，迭代结果时超时抛出TimeoutError。任务异常在迭代时抛出，不影响其他任务执行。
b.超时异常示例
---
from concurrent.futures import ThreadPoolExecutor, TimeoutError
import time
import random
def task_with_delay(task_id):
delay = random.uniform(0.5, 3)
print(f"任务 {task_id} 开始，延迟 {delay:.1f}秒")
time.sleep(delay)
return f"任务-{task_id}-完成"
def task_with_error(task_id):
print(f"任务 {task_id} 执行")
time.sleep(0.5)
if task_id % 3 == 0:
raise ValueError(f"任务 {task_id} 失败")
return f"任务-{task_id}-成功"
with ThreadPoolExecutor(max_workers=5) as executor:
# 示例1: 超时处理
print("=== 超时处理 ===")
task_ids = range(10)
try:
results = executor.map(task_with_delay, task_ids, timeout=2)
for task_id, result in zip(task_ids, results):
print(f"任务 {task_id}: {result}")
except TimeoutError:
print("某个任务超时")
# 示例2: 异常处理
print("\n=== 异常处理 ===")
task_ids = range(10)
results = executor.map(task_with_error, task_ids)
for task_id, result_future in zip(task_ids, results):
try:
result = result_future
print(f"任务 {task_id} 成功: {result}")
except ValueError as e:
print(f"任务 {task_id} 失败: {e}")
except Exception as e:
print(f"任务 {task_id} 异常: {e}")
# 示例3: 部分结果处理
print("\n=== 部分结果处理 ===")
task_ids = range(20)
results = executor.map(task_with_delay, task_ids, timeout=1.5)
successful = []
failed = []
for task_id in task_ids:
try:
result = next(results)
successful.append((task_id, result))
except TimeoutError:
failed.append(task_id)
print(f"任务 {task_id} 超时")
except StopIteration:
break
print(f"成功: {len(successful)}, 失败: {len(failed)}")
---
03.map与submit对比
a.使用场景对比
map适合批量处理同类任务，按顺序返回结果，代码简洁。submit适合异构任务、需要Future对象控制、不关心顺序的场景。
b.对比示例
---
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
def process_item(item_id):
time.sleep(0.5)
return {"id": item_id, "result": item_id * 10}
with ThreadPoolExecutor(max_workers=4) as executor:
items = range(10)
# 方式1: 使用map
print("=== 使用map ===")
start = time.time()
results = list(executor.map(process_item, items))
for result in results:
print(f"Map结果: {result}")
map_time = time.time() - start
print(f"Map耗时: {map_time:.2f}秒\n")
# 方式2: 使用submit
print("=== 使用submit ===")
start = time.time()
futures = [executor.submit(process_item, item) for item in items]
results = [future.result() for future in futures]
for result in results:
print(f"Submit结果: {result}")
submit_time = time.time() - start
print(f"Submit耗时: {submit_time:.2f}秒\n")
# 方式3: submit + as_completed (不保证顺序)
print("=== submit + as_completed ===")
start = time.time()
futures = {executor.submit(process_item, item): item for item in items}
for future in as_completed(futures):
item = futures[future]
result = future.result()
print(f"完成 item {item}: {result}")
as_completed_time = time.time() - start
print(f"as_completed耗时: {as_completed_time:.2f}秒\n")
# 场景对比总结
print("=== 场景选择 ===")
print("map: 批量同类任务，需要顺序结果，代码简洁")
print("submit: 异构任务，需要Future控制，灵活性高")
print("submit+as_completed: 不关心顺序，需要实时处理")
---
4.5 异常处理
01.Future异常捕获
a.result方法异常
Future.result调用时会重新抛出任务中的异常，需要try-except捕获。exception方法返回异常对象不抛出，done为True时可安全调用。
b.异常捕获示例
---
from concurrent.futures import ThreadPoolExecutor
import time
def task_success(task_id):
print(f"任务 {task_id} 执行")
time.sleep(0.5)
return f"任务-{task_id}-成功"
def task_failure(task_id):
print(f"任务 {task_id} 执行")
time.sleep(0.5)
if task_id % 2 == 0:
raise ValueError(f"任务 {task_id} 值错误")
elif task_id % 3 == 0:
raise RuntimeError(f"任务 {task_id} 运行时错误")
return f"任务-{task_id}-成功"
with ThreadPoolExecutor(max_workers=4) as executor:
# 示例1: result方法捕获异常
print("=== result方法捕获异常 ===")
futures = [executor.submit(task_failure, i) for i in range(10)]
for i, future in enumerate(futures):
try:
result = future.result()
print(f"任务 {i} 成功: {result}")
except ValueError as e:
print(f"任务 {i} ValueError: {e}")
except RuntimeError as e:
print(f"任务 {i} RuntimeError: {e}")
except Exception as e:
print(f"任务 {i} 未知异常: {e}")
# 示例2: exception方法获取异常
print("\n=== exception方法获取异常 ===")
futures = [executor.submit(task_failure, i) for i in range(6)]
for i, future in enumerate(futures):
future.result(timeout=2) # 等待完成
for i, future in enumerate(futures):
exc = future.exception()
if exc:
print(f"任务 {i} 异常: {type(exc).__name__}: {exc}")
else:
result = future.result()
print(f"任务 {i} 成功: {result}")
# 示例3: 混合处理
print("\n=== 混合处理 ===")
futures = [executor.submit(task_failure, i) for i in range(8)]
success_count = 0
error_count = 0
for future in futures:
if future.done():
exc = future.exception()
if exc:
error_count += 1
else:
success_count += 1
print(f"成功: {success_count}, 失败: {error_count}")
---
02.批量异常处理
a.as_completed异常处理
as_completed迭代时，每个Future的result调用可能抛出异常。需要在循环内捕获处理，不影响其他任务。
b.批量异常示例
---
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
import random
def unreliable_task(task_id):
print(f"任务 {task_id} 开始")
time.sleep(random.uniform(0.5, 2))
rand = random.random()
if rand < 0.3:
raise ConnectionError(f"任务 {task_id} 连接失败")
elif rand < 0.5:
raise TimeoutError(f"任务 {task_id} 超时")
elif rand < 0.6:
raise ValueError(f"任务 {task_id} 数据错误")
return {"task_id": task_id, "result": f"数据-{task_id}"}
with ThreadPoolExecutor(max_workers=5) as executor:
# 提交任务
futures = {
executor.submit(unreliable_task, i): i
for i in range(20)
}
# 统计结果
results = []
errors = {
"ConnectionError": [],
"TimeoutError": [],
"ValueError": [],
"Other": []
}
# 按完成顺序处理
for future in as_completed(futures):
task_id = futures[future]
try:
result = future.result()
results.append(result)
print(f"任务 {task_id} 成功: {result}")
except ConnectionError as e:
errors["ConnectionError"].append(task_id)
print(f"任务 {task_id} 连接错误: {e}")
except TimeoutError as e:
errors["TimeoutError"].append(task_id)
print(f"任务 {task_id} 超时: {e}")
except ValueError as e:
errors["ValueError"].append(task_id)
print(f"任务 {task_id} 值错误: {e}")
except Exception as e:
errors["Other"].append(task_id)
print(f"任务 {task_id} 其他错误: {e}")
# 输出统计
print(f"\n=== 执行统计 ===")
print(f"成功: {len(results)}")
for error_type, task_ids in errors.items():
if task_ids:
print(f"{error_type}: {len(task_ids)} 个 - {task_ids}")
---
03.异常重试机制
a.重试策略
任务失败时可实现重试机制，设置最大重试次数和重试延迟。记录失败原因，达到最大次数后放弃或记录错误。
b.重试机制示例
---
from concurrent.futures import ThreadPoolExecutor
import time
import random
def flaky_task(task_id, attempt=1):
print(f"任务 {task_id} 尝试 {attempt}")
time.sleep(0.5)
# 30%概率失败
if random.random() < 0.3:
raise RuntimeError(f"任务 {task_id} 临时失败")
return f"任务-{task_id}-成功"
def retry_task(executor, func, task_id, max_retries=3, delay=1):
for attempt in range(1, max_retries + 1):
try:
future = executor.submit(func, task_id, attempt)
result = future.result()
print(f"任务 {task_id} 成功 (尝试 {attempt})")
return result
except Exception as e:
print(f"任务 {task_id} 失败 (尝试 {attempt}): {e}")
if attempt < max_retries:
print(f"任务 {task_id} 等待 {delay} 秒后重试")
time.sleep(delay)
else:
print(f"任务 {task_id} 达到最大重试次数")
raise
with ThreadPoolExecutor(max_workers=4) as executor:
# 示例1: 基本重试
print("=== 基本重试机制 ===")
task_ids = range(5)
for task_id in task_ids:
try:
result = retry_task(executor, flaky_task, task_id, max_retries=3, delay=0.5)
print(f"最终结果: {result}\n")
except Exception as e:
print(f"任务 {task_id} 最终失败: {e}\n")
# 示例2: 批量重试
print("\n=== 批量重试 ===")
def batch_retry(task_ids, max_retries=3):
results = {}
failed = {}
for task_id in task_ids:
try:
result = retry_task(executor, flaky_task, task_id, max_retries, delay=0.3)
results[task_id] = result
except Exception as e:
failed[task_id] = str(e)
return results, failed
task_ids = range(10)
results, failed = batch_retry(task_ids, max_retries=3)
print(f"\n成功: {len(results)} 个")
print(f"失败: {len(failed)} 个")
if failed:
print(f"失败任务: {list(failed.keys())}")
# 示例3: 指数退避重试
print("\n=== 指数退避重试 ===")
def exponential_backoff_retry(executor, func, task_id, max_retries=4):
for attempt in range(1, max_retries + 1):
try:
future = executor.submit(func, task_id, attempt)
result = future.result()
return result
except Exception as e:
if attempt < max_retries:
delay = 2 ** (attempt - 1) # 1, 2, 4, 8秒
print(f"任务 {task_id} 失败，等待 {delay} 秒")
time.sleep(delay)
else:
raise
for task_id in range(3):
try:
result = exponential_backoff_retry(executor, flaky_task, task_id)
print(f"任务 {task_id} 最终成功: {result}\n")
except Exception as e:
print(f"任务 {task_id} 最终失败: {e}\n")
---
5. 同步原语
5.1 Lock与RLock
01.Lock互斥锁
a.Lock基础
Lock提供互斥访问，同一时刻只有一个线程持有锁。acquire获取锁会阻塞，release释放锁。使用with语句自动管理锁的获取和释放。
b.Lock使用示例
---
import threading
import time
counter = 0
lock = threading.Lock()
def increment_without_lock(n):
global counter
for _ in range(n):
temp = counter
time.sleep(0.0001)
counter = temp + 1
def increment_with_lock(n):
global counter
for _ in range(n):
with lock:
temp = counter
time.sleep(0.0001)
counter = temp + 1
# 不使用锁的竞态条件
print("=== 不使用锁 ===")
counter = 0
threads = [threading.Thread(target=increment_without_lock, args=(100,)) for _ in range(3)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"预期: 300, 实际: {counter}")
# 使用锁保护
print("\n=== 使用Lock ===")
counter = 0
threads = [threading.Thread(target=increment_with_lock, args=(100,)) for _ in range(3)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"预期: 300, 实际: {counter}")
# 手动管理锁
print("\n=== 手动管理Lock ===")
lock = threading.Lock()
shared_list = []
def add_items(items):
lock.acquire()
try:
for item in items:
shared_list.append(item)
time.sleep(0.01)
finally:
lock.release()
t1 = threading.Thread(target=add_items, args=([1, 2, 3],))
t2 = threading.Thread(target=add_items, args=([4, 5, 6],))
t1.start()
t2.start()
t1.join()
t2.join()
print(f"共享列表: {shared_list}")
---
02.RLock可重入锁
a.RLock特性
RLock允许同一线程多次获取锁，需要相同次数的release。适合递归函数或需要多次获取锁的场景。不同线程间仍然互斥。
b.RLock使用示例
---
import threading
import time
class BankAccount:
def __init__(self, balance):
self.balance = balance
self.lock = threading.RLock()
def deposit(self, amount):
with self.lock:
print(f"存款 {amount}")
self.balance += amount
self._log_transaction("deposit", amount)
def withdraw(self, amount):
with self.lock:
if self.balance >= amount:
print(f"取款 {amount}")
self.balance -= amount
self._log_transaction("withdraw", amount)
return True
return False
def _log_transaction(self, type, amount):
with self.lock: # 可重入
print(f"记录: {type} {amount}, 余额: {self.balance}")
def transfer_to(self, other, amount):
with self.lock:
if self.withdraw(amount):
other.deposit(amount)
print(f"转账 {amount} 成功")
account1 = BankAccount(1000)
account2 = BankAccount(500)
def do_transactions():
account1.deposit(200)
account1.withdraw(100)
account1.transfer_to(account2, 300)
threads = [threading.Thread(target=do_transactions) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"\n账户1余额: {account1.balance}")
print(f"账户2余额: {account2.balance}")
---
03.Lock与RLock对比
a.使用场景对比
Lock适合简单互斥场景，性能更好。RLock适合递归调用、嵌套锁场景，同一���程可多次获取。Lock不可重入会导致死锁。
b.对比示例
---
import threading
import time
# Lock死锁示例
print("=== Lock不可重入导致死锁 ===")
lock = threading.Lock()
def recursive_with_lock(n):
if n <= 0:
return
lock.acquire()
print(f"Lock获取: {n}")
time.sleep(0.1)
if n > 1:
recursive_with_lock(n - 1) # 死锁
lock.release()
# 注释掉避免死锁
# t = threading.Thread(target=recursive_with_lock, args=(3,))
# t.start()
# t.join(timeout=2)
print("Lock递归会死锁（已注释）")
# RLock可重入
print("\n=== RLock可重入 ===")
rlock = threading.RLock()
def recursive_with_rlock(n):
if n <= 0:
return
rlock.acquire()
print(f"RLock获取: {n}")
time.sleep(0.1)
if n > 1:
recursive_with_rlock(n - 1)
rlock.release()
t = threading.Thread(target=recursive_with_rlock, args=(3,))
t.start()
t.join()
print("RLock递归成功")
# 性能对比
print("\n=== 性能对比 ===")
lock = threading.Lock()
rlock = threading.RLock()
iterations = 100000
def test_lock():
for _ in range(iterations):
with lock:
pass
def test_rlock():
for _ in range(iterations):
with rlock:
pass
start = time.time()
test_lock()
lock_time = time.time() - start
start = time.time()
test_rlock()
rlock_time = time.time() - start
print(f"Lock耗时: {lock_time:.4f}秒")
print(f"RLock耗时: {rlock_time:.4f}秒")
print(f"RLock开销: {(rlock_time/lock_time - 1)*100:.1f}%")
---
5.2 Semaphore与BoundedSemaphore
01.Semaphore信号量
a.Semaphore基础
Semaphore维护内部计数器，acquire减1，release加1。计数器为0时acquire阻塞。用于限制同时访问资源的线程数，如连接池、并发请求限制。
b.Semaphore示例
---
import threading
import time
import random
# 限制并发数
semaphore = threading.Semaphore(3)
def access_resource(worker_id):
print(f"Worker {worker_id} 等待访问")
with semaphore:
print(f"Worker {worker_id} 获得访问权限")
time.sleep(random.uniform(1, 3))
print(f"Worker {worker_id} 释放访问权限")
threads = [threading.Thread(target=access_resource, args=(i,)) for i in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
# 连接池模拟
print("\n=== 连接池示例 ===")
class ConnectionPool:
def __init__(self, max_connections):
self.semaphore = threading.Semaphore(max_connections)
self.connections = []
def acquire_connection(self, client_id):
self.semaphore.acquire()
conn = f"Connection-{client_id}"
self.connections.append(conn)
print(f"客户端 {client_id} 获取连接: {conn}")
return conn
def release_connection(self, conn, client_id):
self.connections.remove(conn)
print(f"客户端 {client_id} 释放连接: {conn}")
self.semaphore.release()
pool = ConnectionPool(5)
def use_connection(client_id):
conn = pool.acquire_connection(client_id)
time.sleep(random.uniform(0.5, 2))
pool.release_connection(conn, client_id)
threads = [threading.Thread(target=use_connection, args=(i,)) for i in range(15)]
for t in threads:
t.start()
for t in threads:
t.join()
---
02.BoundedSemaphore有界信号量
a.BoundedSemaphore特性
BoundedSemaphore是Semaphore的子类，release时检查计数器不超过初始值。防止release次数多于acquire导致计数器无限增长。
b.BoundedSemaphore示例
---
import threading
import time
# Semaphore可能被过度释放
print("=== Semaphore过度释放 ===")
sem = threading.Semaphore(2)
sem.release()
sem.release()
print(f"Semaphore过度释放成功，计数器变为4")
# BoundedSemaphore防止过度释放
print("\n=== BoundedSemaphore防止过度释放 ===")
bounded_sem = threading.BoundedSemaphore(2)
try:
bounded_sem.release()
except ValueError as e:
print(f"BoundedSemaphore错误: {e}")
# 实际应用
print("\n=== 资源池管理 ===")
class ResourcePool:
def __init__(self, size):
self.semaphore = threading.BoundedSemaphore(size)
self.resources = [f"Resource-{i}" for i in range(size)]
self.lock = threading.Lock()
def acquire(self, worker_id):
self.semaphore.acquire()
with self.lock:
resource = self.resources.pop()
print(f"Worker {worker_id} 获取 {resource}")
return resource
def release(self, resource, worker_id):
with self.lock:
self.resources.append(resource)
print(f"Worker {worker_id} 释放 {resource}")
self.semaphore.release()
pool = ResourcePool(3)
def worker(worker_id):
resource = pool.acquire(worker_id)
time.sleep(1)
pool.release(resource, worker_id)
threads = [threading.Thread(target=worker, args=(i,)) for i in range(8)]
for t in threads:
t.start()
for t in threads:
t.join()
---
03.信号量应用场景
a.限流与并发控制
信号量常用于限制并发请求数、控制资源访问、实现限流器。可动态调整并发度，适合API限流、数据库连接池等场景。
b.应用场景示例
---
import threading
import time
import random
# API限流器
print("=== API限流器 ===")
class RateLimiter:
def __init__(self, max_concurrent):
self.semaphore = threading.Semaphore(max_concurrent)
def __enter__(self):
self.semaphore.acquire()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.semaphore.release()
limiter = RateLimiter(5)
def api_request(request_id):
with limiter:
print(f"请求 {request_id} 开始")
time.sleep(random.uniform(0.5, 1.5))
print(f"请求 {request_id} 完成")
threads = [threading.Thread(target=api_request, args=(i,)) for i in range(20)]
for t in threads:
t.start()
for t in threads:
t.join()
# 生产者消费者限流
print("\n=== 生产者消费者限流 ===")
buffer = []
buffer_lock = threading.Lock()
empty_slots = threading.Semaphore(10)
filled_slots = threading.Semaphore(0)
def producer(producer_id):
for i in range(5):
item = f"P{producer_id}-Item{i}"
empty_slots.acquire()
with buffer_lock:
buffer.append(item)
print(f"生产者 {producer_id} 生产: {item}, 缓冲区: {len(buffer)}")
filled_slots.release()
time.sleep(random.uniform(0.1, 0.3))
def consumer(consumer_id):
for _ in range(5):
filled_slots.acquire()
with buffer_lock:
item = buffer.pop(0)
print(f"消费者 {consumer_id} 消费: {item}, 缓冲区: {len(buffer)}")
empty_slots.release()
time.sleep(random.uniform(0.2, 0.5))
producers = [threading.Thread(target=producer, args=(i,)) for i in range(2)]
consumers = [threading.Thread(target=consumer, args=(i,)) for i in range(2)]
for t in producers + consumers:
t.start()
for t in producers + consumers:
t.join()
---
5.3 Event与Condition
01.Event事件对象
a.Event基础
Event内部维护标志位，set设置为True，clear设置为False，wait阻塞直到标志为True。用于线程间简单信号通知，如启动信号、停止信号。
b.Event示例
---
import threading
import time
event = threading.Event()
def waiter(name):
print(f"{name} 等待事件")
event.wait()
print(f"{name} 收到事件，开始工作")
time.sleep(1)
print(f"{name} 完成工作")
def setter():
print("Setter 准备中...")
time.sleep(2)
print("Setter 触发事件")
event.set()
threads = [threading.Thread(target=waiter, args=(f"Worker-{i}",)) for i in range(3)]
threads.append(threading.Thread(target=setter))
for t in threads:
t.start()
for t in threads:
t.join()
# 超时等待
print("\n=== 超时等待 ===")
event2 = threading.Event()
def timeout_waiter():
print("等待事件（超时3秒）")
result = event2.wait(timeout=3)
if result:
print("收到事件")
else:
print("等待超时")
t = threading.Thread(target=timeout_waiter)
t.start()
t.join()
# 重复使用Event
print("\n=== 重复使用Event ===")
event3 = threading.Event()
def worker(worker_id):
for i in range(3):
event3.wait()
print(f"Worker {worker_id} 执行任务 {i}")
time.sleep(0.5)
def controller():
for i in range(3):
time.sleep(1)
print(f"控制器触发第 {i+1} 次")
event3.set()
time.sleep(0.1)
event3.clear()
workers = [threading.Thread(target=worker, args=(i,)) for i in range(2)]
ctrl = threading.Thread(target=controller)
for t in workers + [ctrl]:
t.start()
for t in workers + [ctrl]:
t.join()
---
02.Condition条件变量
a.Condition基础
Condition结合Lock和通知机制，wait释放锁并阻塞，notify唤醒一个等待线程，notify_all唤醒所有等待线程。用于复杂的线程协调。
b.Condition示例
---
import threading
import time
import random
condition = threading.Condition()
items = []
def producer(producer_id):
for i in range(5):
time.sleep(random.uniform(0.5, 1.5))
with condition:
item = f"P{producer_id}-Item{i}"
items.append(item)
print(f"生产者 {producer_id} 生产: {item}, 总数: {len(items)}")
condition.notify()
def consumer(consumer_id):
for _ in range(5):
with condition:
while not items:
print(f"消费者 {consumer_id} 等待")
condition.wait()
item = items.pop(0)
print(f"消费者 {consumer_id} 消费: {item}, 剩余: {len(items)}")
producers = [threading.Thread(target=producer, args=(i,)) for i in range(2)]
consumers = [threading.Thread(target=consumer, args=(i,)) for i in range(2)]
for t in producers + consumers:
t.start()
for t in producers + consumers:
t.join()
# notify_all示例
print("\n=== notify_all示例 ===")
condition2 = threading.Condition()
ready = False
def worker(worker_id):
with condition2:
while not ready:
print(f"Worker {worker_id} 等待")
condition2.wait()
print(f"Worker {worker_id} 开始工作")
def start_all():
global ready
time.sleep(2)
with condition2:
ready = True
print("通知所有Worker")
condition2.notify_all()
workers = [threading.Thread(target=worker, args=(i,)) for i in range(5)]
starter = threading.Thread(target=start_all)
for t in workers + [starter]:
t.start()
for t in workers + [starter]:
t.join()
---
03.Event与Condition对比
a.使用场景对比
Event适合简单的开关信号，所有等待线程同时唤醒。Condition适合复杂条件等待，可选择唤醒一个或全部线程，支持条件判断。
b.对比示例
---
import threading
import time
# Event实现启动屏障
print("=== Event启动屏障 ===")
start_event = threading.Event()
def event_worker(worker_id):
print(f"Worker {worker_id} 准备就绪")
start_event.wait()
print(f"Worker {worker_id} 开始执行")
workers = [threading.Thread(target=event_worker, args=(i,)) for i in range(3)]
for t in workers:
t.start()
time.sleep(1)
print("启动所有Worker")
start_event.set()
for t in workers:
t.join()
# Condition实现阶段同步
print("\n=== Condition阶段同步 ===")
condition = threading.Condition()
phase = 0
def condition_worker(worker_id):
global phase
for expected_phase in range(3):
with condition:
while phase != expected_phase:
condition.wait()
print(f"Worker {worker_id} 执行阶段 {phase}")
time.sleep(0.5)
def phase_controller():
global phase
for p in range(3):
time.sleep(1)
with condition:
phase = p
print(f"进入阶段 {phase}")
condition.notify_all()
workers = [threading.Thread(target=condition_worker, args=(i,)) for i in range(3)]
controller = threading.Thread(target=phase_controller)
for t in workers + [controller]:
t.start()
for t in workers + [controller]:
t.join()
# 性能对比
print("\n=== 性能对比 ===")
event = threading.Event()
condition = threading.Condition()
def test_event():
for _ in range(1000):
event.set()
event.clear()
def test_condition():
for _ in range(1000):
with condition:
condition.notify_all()
start = time.time()
test_event()
event_time = time.time() - start
start = time.time()
test_condition()
condition_time = time.time() - start
print(f"Event耗时: {event_time:.4f}秒")
print(f"Condition耗时: {condition_time:.4f}秒")
---
5.4 Barrier屏障
01.Barrier基础
a.Barrier原理
Barrier设置parties数量，wait阻塞直到parties个线程都调用wait，然后同时释放所有线程。用于多阶段任务同步，确保所有线程完成当前阶段再进入下一阶段。
b.Barrier基础示例
---
import threading
import time
import random
barrier = threading.Barrier(3)
def worker(worker_id):
print(f"Worker {worker_id} 阶段1开始")
time.sleep(random.uniform(1, 3))
print(f"Worker {worker_id} 阶段1完成，等待其他线程")
barrier.wait()
print(f"Worker {worker_id} 阶段2开始")
time.sleep(random.uniform(1, 2))
print(f"Worker {worker_id} 阶段2完成")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join()
# 多阶段同步
print("\n=== 多阶段同步 ===")
barrier2 = threading.Barrier(4)
def multi_phase_worker(worker_id):
for phase in range(3):
print(f"Worker {worker_id} 执行阶段 {phase+1}")
time.sleep(random.uniform(0.5, 1.5))
print(f"Worker {worker_id} 完成阶段 {phase+1}，等待")
barrier2.wait()
print(f"Worker {worker_id} 进入下一阶段")
threads = [threading.Thread(target=multi_phase_worker, args=(i,)) for i in range(4)]
for t in threads:
t.start()
for t in threads:
t.join()
---
02.Barrier回调与重置
a.action回调
Barrier支持action参数，最后一个到达的线程执行action回调。reset重置Barrier，abort中止等待并抛出BrokenBarrierError。
b.回调重置示例
---
import threading
import time
def phase_complete():
print("=== 所有线程完成当前阶段 ===")
barrier = threading.Barrier(3, action=phase_complete)
def worker_with_callback(worker_id):
for phase in range(2):
print(f"Worker {worker_id} 阶段 {phase+1}")
time.sleep(1)
barrier.wait()
threads = [threading.Thread(target=worker_with_callback, args=(i,)) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join()
# reset示例
print("\n=== reset示例 ===")
barrier2 = threading.Barrier(2)
def worker_reset(worker_id):
try:
print(f"Worker {worker_id} 等待")
barrier2.wait(timeout=2)
print(f"Worker {worker_id} 通过屏障")
except threading.BrokenBarrierError:
print(f"Worker {worker_id} 屏障被破坏")
t1 = threading.Thread(target=worker_reset, args=(1,))
t1.start()
time.sleep(0.5)
print("重置屏障")
barrier2.reset()
t2 = threading.Thread(target=worker_reset, args=(2,))
t2.start()
t1.join()
t2.join()
# abort示例
print("\n=== abort示例 ===")
barrier3 = threading.Barrier(3)
def worker_abort(worker_id):
try:
print(f"Worker {worker_id} 等待")
barrier3.wait()
print(f"Worker {worker_id} 通过")
except threading.BrokenBarrierError:
print(f"Worker {worker_id} 屏障中止")
threads = [threading.Thread(target=worker_abort, args=(i,)) for i in range(3)]
for t in threads:
t.start()
time.sleep(0.5)
print("中止屏障")
barrier3.abort()
for t in threads:
t.join()
---
03.Barrier应用场景
a.并行计算同步
Barrier适合并行计算中的阶段同步，如MapReduce、迭代算法、分布式计算。确保所有worker完成当前迭代再进入下一迭代。
b.应用场景示例
---
import threading
import time
import random
# 并行矩阵计算
print("=== 并行矩阵计算 ===")
matrix = [[random.randint(1, 10) for _ in range(5)] for _ in range(5)]
result = [[0] * 5 for _ in range(5)]
barrier = threading.Barrier(5)
def compute_row(row_id):
for iteration in range(3):
print(f"行 {row_id} 迭代 {iteration+1}")
for col in range(5):
result[row_id][col] = matrix[row_id][col] * 2
time.sleep(0.1)
barrier.wait()
if row_id == 0:
print(f"迭代 {iteration+1} 完成")
threads = [threading.Thread(target=compute_row, args=(i,)) for i in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
# 模拟训练同步
print("\n=== 模拟训练同步 ===")
workers = 4
barrier = threading.Barrier(workers)
global_model = {"weights": 0}
lock = threading.Lock()
def train_worker(worker_id):
local_gradient = 0
for epoch in range(3):
local_gradient = random.randint(1, 10)
print(f"Worker {worker_id} Epoch {epoch+1} 梯度: {local_gradient}")
time.sleep(random.uniform(0.5, 1))
with lock:
global_model["weights"] += local_gradient
barrier.wait()
if worker_id == 0:
print(f"Epoch {epoch+1} 全局权重: {global_model['weights']}")
threads = [threading.Thread(target=train_worker, args=(i,)) for i in range(workers)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"最终权重: {global_model['weights']}")
---
5.5 Queue队列
01.Queue基础
a.Queue类型
Queue提供FIFO队列，LifoQueue提供LIFO栈，PriorityQueue提供优先级队列。put添加元素，get获取元素，支持阻塞和超时。线程安全。
b.Queue基础示例
---
import threading
import queue
import time
q = queue.Queue()
def producer(producer_id):
for i in range(5):
item = f"P{producer_id}-Item{i}"
q.put(item)
print(f"生产者 {producer_id} 生产: {item}")
time.sleep(0.5)
def consumer(consumer_id):
while True:
try:
item = q.get(timeout=3)
print(f"消费者 {consumer_id} 消费: {item}")
q.task_done()
time.sleep(1)
except queue.Empty:
print(f"消费者 {consumer_id} 队列空，退出")
break
producers = [threading.Thread(target=producer, args=(i,)) for i in range(2)]
consumers = [threading.Thread(target=consumer, args=(i,)) for i in range(2)]
for t in producers + consumers:
t.start()
for t in producers + consumers:
t.join()
# LifoQueue示例
print("\n=== LifoQueue ===")
lifo_q = queue.LifoQueue()
for i in range(5):
lifo_q.put(i)
while not lifo_q.empty():
print(f"LIFO: {lifo_q.get()}")
# PriorityQueue示例
print("\n=== PriorityQueue ===")
pq = queue.PriorityQueue()
pq.put((3, "低优先级"))
pq.put((1, "高优先级"))
pq.put((2, "中优先级"))
while not pq.empty():
priority, item = pq.get()
print(f"优先级 {priority}: {item}")
---
02.Queue高级特性
a.maxsize与阻塞
Queue支持maxsize限制大小，满时put阻塞。join等待所有task_done调用。qsize、empty、full查询状态，但非线程安全。
b.高级特性示例
---
import threading
import queue
import time
# 有界队列
print("=== 有界队列 ===")
bounded_q = queue.Queue(maxsize=3)
def bounded_producer():
for i in range(10):
print(f"尝试放入 {i}")
bounded_q.put(i)
print(f"成功放入 {i}, 队列大小: {bounded_q.qsize()}")
time.sleep(0.5)
def bounded_consumer():
time.sleep(2)
for _ in range(10):
item = bounded_q.get()
print(f"取出 {item}")
time.sleep(1)
t1 = threading.Thread(target=bounded_producer)
t2 = threading.Thread(target=bounded_consumer)
t1.start()
t2.start()
t1.join()
t2.join()
# join等待
print("\n=== join等待 ===")
task_q = queue.Queue()
def worker():
while True:
item = task_q.get()
if item is None:
break
print(f"处理: {item}")
time.sleep(0.5)
task_q.task_done()
threads = [threading.Thread(target=worker) for _ in range(3)]
for t in threads:
t.start()
for i in range(10):
task_q.put(f"Task-{i}")
task_q.join()
print("所有任务完成")
for _ in range(3):
task_q.put(None)
for t in threads:
t.join()
---
03.Queue应用模式
a.生产者消费者模式
Queue是生产者消费者模式的标准实现，解耦生产和消费，支持多生产者多消费者，自动处理同步和缓冲。
b.应用模式示例
---
import threading
import queue
import time
import random
# 任务队列模式
print("=== 任务队列模式 ===")
task_queue = queue.Queue()
result_queue = queue.Queue()
def task_producer():
for i in range(20):
task = {"id": i, "data": random.randint(1, 100)}
task_queue.put(task)
print(f"生成任务 {i}")
time.sleep(0.1)
def task_worker(worker_id):
while True:
try:
task = task_queue.get(timeout=2)
print(f"Worker {worker_id} 处理任务 {task['id']}")
time.sleep(random.uniform(0.3, 0.8))
result = {"task_id": task["id"], "result": task["data"] * 2}
result_queue.put(result)
task_queue.task_done()
except queue.Empty:
break
def result_collector():
results = []
while True:
try:
result = result_queue.get(timeout=3)
results.append(result)
print(f"收集结果: 任务 {result['task_id']}")
except queue.Empty:
break
print(f"总共收集 {len(results)} 个结果")
producer = threading.Thread(target=task_producer)
workers = [threading.Thread(target=task_worker, args=(i,)) for i in range(4)]
collector = threading.Thread(target=result_collector)
producer.start()
for w in workers:
w.start()
producer.join()
for w in workers:
w.join()
collector.start()
collector.join()
# 优先级任务调度
print("\n=== 优先级任务调度 ===")
priority_q = queue.PriorityQueue()
def priority_producer():
tasks = [
(3, "低优先级任务1"),
(1, "高优先级任务1"),
(2, "中优先级任务1"),
(1, "高优先级任务2"),
(3, "低优先级任务2")
]
for priority, task in tasks:
priority_q.put((priority, task))
print(f"添加任务: {task}")
def priority_worker():
while not priority_q.empty():
try:
priority, task = priority_q.get(timeout=1)
print(f"执行: {task} (优先级 {priority})")
time.sleep(0.5)
except queue.Empty:
break
producer = threading.Thread(target=priority_producer)
workers = [threading.Thread(target=priority_worker) for _ in range(2)]
producer.start()
producer.join()
for w in workers:
w.start()
for w in workers:
w.join()
---
6. GIL与性能优化
6.1 全局解释器锁GIL
01.GIL基础概念
a.GIL原理
GIL是CPython解释器的互斥锁，确保同一时刻只有一个线程执行Python字节码。保护Python对象的内存管理，防止多线程访问导致问题。线程在执行期间获取GIL。
b.GIL影响示例
---
import threading
import time
import sys
# CPU密集型任务
def cpu_task(n):
result = 0
for i in range(n):
result += i * i
return result
# 检查GIL状态
def check_gil():
if hasattr(sys, 'getswitchinterval'):
interval = sys.getswitchinterval()
print(f"线程切换间隔: {interval}秒")
# 单线程执行
print("=== 单线程执行 ===")
start = time.time()
result1 = cpu_task(1000000)
result2 = cpu_task(1000000)
single_time = time.time() - start
print(f"单线程耗时: {single_time:.2f}秒")
# 多线程执行
print("\n=== 多线程执行 ===")
start = time.time()
t1 = threading.Thread(target=lambda: cpu_task(1000000))
t2 = threading.Thread(target=lambda: cpu_task(1000000))
t1.start()
t2.start()
t1.join()
t2.join()
multi_time = time.time() - start
print(f"多线程耗时: {multi_time:.2f}秒")
print(f"性能提升: {single_time/multi_time:.2f}x")
check_gil()
---
02.GIL获取释放
a.GIL机制
Python解释器周期性检查是否有线程需要运行，每100个字节码或时间片到期时释放GIL。IO操作期间自动释放GIL，允许其他线程运行。
b.GIL获取释放示例
---
import threading
import time
import sys
def io_task():
print(f"IO任务开始，线程: {threading.current_thread().name}")
time.sleep(2) # IO操作，释放GIL
print(f"IO任务完成，线程: {threading.current_thread().name}")
def cpu_task():
print(f"CPU任务开始，线程: {threading.current_thread().name}")
count = 0
end_time = time.time() + 2
while time.time() < end_time: # CPU密集型，持有GIL
count += 1
print(f"CPU任务完成，线程: {threading.current_thread().name}, 计数: {count}")
# CPU + IO混合
print("=== CPU + IO混合 ===")
start = time.time()
io_thread = threading.Thread(target=io_task, name="IO-Thread")
cpu_thread = threading.Thread(target=cpu_task, name="CPU-Thread")
cpu_thread.start()
time.sleep(0.1) # 确保CPU线程先开始
io_thread.start()
cpu_thread.join()
io_thread.join()
total_time = time.time() - start
print(f"总耗时: {total_time:.2f}秒")
# 多个IO任务
print("\n=== 多个IO任务 ===")
start = time.time()
io_tasks = [threading.Thread(target=io_task, name=f"IO-{i}") for i in range(3)]
for task in io_tasks:
task.start()
for task in io_tasks:
task.join()
io_time = time.time() - start
print(f"IO并发耗时: {io_time:.2f}秒")
# 查看线程切换间隔
print(f"\n线程切换间隔: {sys.getswitchinterval()}秒")
---
03.GIL绕过策略
a.绕过GIL方法
使用多进程、C扩展模块、ctypes调用C代码、JIT编译器等方式绕过GIL限制。GIL仅影响CPython，其他Python实现可能没有GIL。
b.GIL绕过示例
---
import threading
import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import os
def cpu_intensive_task(n):
result = 0
for i in range(n):
result += i * i * i
return result
# 线程池受GIL影响
print("=== 线程池执行 ===")
with ThreadPoolExecutor(max_workers=4) as executor:
start = time.time()
futures = [executor.submit(cpu_intensive_task, 500000) for _ in range(4)]
results = [f.result() for f in futures]
thread_time = time.time() - start
print(f"线程池耗时: {thread_time:.2f}秒")
# 进程池绕过GIL
print("\n=== 进程池执行 ===")
if __name__ == '__main__':
with ProcessPoolExecutor(max_workers=4) as executor:
start = time.time()
futures = [executor.submit(cpu_intensive_task, 500000) for _ in range(4)]
results = [f.result() for f in futures]
process_time = time.time() - start
print(f"进程池耗时: {process_time:.2f}秒")
print(f"性能提升: {thread_time/process_time:.2f}x")
# 检查并行性
print(f"\nCPU核心数: {os.cpu_count()}")
print(f"线程池: 受GIL限制，CPU密集型任务无法真正并行")
print(f"进程池: 每个进程独立GIL，可实现真正并行")
---
6.2 CPU密集型任务优化
01.多进程优化
a.CPU密集型特点
CPU密集型任务受GIL限制，多线程无法真正并行。使用多进程绕过GIL，每个进程独立GIL，可充分利用多核CPU。适合数学计算、数据处理等任务。
b.多进程优化示例
---
import multiprocessing
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
import os
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
def prime_check(n):
if n < 2:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True
def cpu_intensive_task(task_id):
print(f"进程 {os.getpid()} 执行任务 {task_id}")
result = fibonacci(35)
primes = sum(1 for i in range(2, 10000) if prime_check(i))
return {"task_id": task_id, "fib": result, "primes": primes}
# 单线程基准
print("=== 单线程基准 ===")
start = time.time()
results = [cpu_intensive_task(i) for i in range(4)]
single_time = time.time() - start
print(f"单线程耗时: {single_time:.2f}秒")
# 多线程（受GIL限制）
print("\n=== 多线程执行 ===")
with ThreadPoolExecutor(max_workers=4) as executor:
start = time.time()
results = list(executor.map(cpu_intensive_task, range(4)))
thread_time = time.time() - start
print(f"多线程耗时: {thread_time:.2f}秒")
# 多进程（绕过GIL）
print("\n=== 多进程执行 ===")
if __name__ == '__main__':
with ProcessPoolExecutor(max_workers=4) as executor:
start = time.time()
results = list(executor.map(cpu_intensive_task, range(4)))
process_time = time.time() - start
print(f"多进程耗时: {process_time:.2f}秒")
print(f"性能提升: {single_time/process_time:.2f}x")
for result in results:
print(f"任务 {result['task_id']}: fib={result['fib']}, primes={result['primes']}")
---
02.进程池优化
a.进程池配置
合理配置进程数量（通常等于CPU核心数），减少进程创建销毁开销。使用进程池重用进程，支持结果缓存和负载均衡。
b.进程池优化示例
---
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
import time
import math
def matrix_multiply_chunk(chunk_info):
chunk_id, matrices_a, matrices_b, start_idx, end_idx = chunk_info
results = []
for i in range(start_idx, end_idx):
a, b = matrices_a[i], matrices_b[i]
# 矩阵乘法
result = [[a[i][k] * b[k][j] for k in range(len(a[0]))] for j in range(len(b[0]))]
results.append(result)
return chunk_id, results
def generate_matrices(count, size):
import random
return [random.randint(1, 10) for _ in range(count)], [random.randint(1, 10) for _ in range(count)]
# 优化的进程池使用
print("=== 进程池矩阵计算 ===")
if __name__ == '__main__':
# 配置参数
num_matrices = 20
matrix_size = 100
chunk_size = 5
num_processes = multiprocessing.cpu_count()
print(f"CPU核心数: {num_processes}")
print(f"矩阵数量: {num_matrices}, 每块大小: {chunk_size}")
# 分割任务
matrices_a = [list(range(matrix_size * matrix_size)) for _ in range(num_matrices)]
matrices_b = [list(range(matrix_size * matrix_size)) for _ in range(num_matrices)]
chunks = []
for i in range(0, num_matrices, chunk_size):
chunk_info = (i//chunk_size, matrices_a, matrices_b, i, min(i+chunk_size, num_matrices))
chunks.append(chunk_info)
# 使用进程池
with ProcessPoolExecutor(max_workers=num_processes) as executor:
start = time.time()
results = list(executor.map(matrix_multiply_chunk, chunks))
pool_time = time.time() - start
print(f"进程池耗时: {pool_time:.2f}秒")
# 单进程对比
start = time.time()
single_result = matrix_multiply_chunk((0, matrices_a, matrices_b, 0, num_matrices))
single_time = time.time() - start
print(f"单进程耗时: {single_time:.2f}秒")
print(f"性能提升: {single_time/pool_time:.2f}x")
print(f"处理了 {len(results)} 个数据块")
---
03.任务分解优化
a.任务分解策略
将大任务分解为适当大小的子任务，避免负载不均。考虑通信开销，平衡粒度和并行度。使用分而治之、MapReduce等策略。
b.任务分解示例
---
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
import time
import random
def process_data_chunk(chunk):
chunk_id, data = chunk
print(f"处理数据块 {chunk_id}，大小: {len(data)}")
# 模拟复杂处理
result = sum(x * x for x in data if x % 2 == 0)
return chunk_id, result
def naive_chunking(data, num_chunks):
chunk_size = len(data) // num_chunks
chunks = []
for i in range(num_chunks):
start = i * chunk_size
end = (i + 1) * chunk_size if i < num_chunks - 1 else len(data)
chunks.append((i, data[start:end]))
return chunks
def smart_chunking(data, num_chunks):
# 根据数据特征智能分块
data_groups = []
for i in range(0, len(data), 1000): # 每1000个数据一组
chunk_data = data[i:i+1000]
avg_value = sum(chunk_data) / len(chunk_data)
data_groups.append((i//1000, chunk_data, avg_value))
# 合并小的组
optimized_chunks = []
current_chunk = []
current_size = 0
for group in data_groups:
chunk_id, group_data, avg_value = group
if len(group_data) > 500: # 大组单独处理
if current_chunk:
optimized_chunks.append((len(optimized_chunks), current_chunk))
current_chunk = []
current_size = 0
optimized_chunks.append((len(optimized_chunks), group_data))
else:
current_chunk.extend(group_data)
current_size += len(group_data)
if current_size >= 1000:
optimized_chunks.append((len(optimized_chunks), current_chunk))
current_chunk = []
current_size = 0
if current_chunk:
optimized_chunks.append((len(optimized_chunks), current_chunk))
return optimized_chunks
# 测试不同分块策略
print("=== 任务分解优化 ===")
if __name__ == '__main__':
# 生成测试数据
data_size = 100000
test_data = [random.randint(1, 100) for _ in range(data_size)]
num_workers = multiprocessing.cpu_count()
# 简单分块
print(f"\n简单分块策略:")
naive_chunks = naive_chunking(test_data, num_workers)
chunk_sizes = [len(chunk[1]) for chunk in naive_chunks]
print(f"数据块大小: {chunk_sizes}")
with ProcessPoolExecutor(max_workers=num_workers) as executor:
start = time.time()
naive_results = list(executor.map(process_data_chunk, naive_chunks))
naive_time = time.time() - start
print(f"简单分块耗时: {naive_time:.2f}秒")
# 智能分块
print(f"\n智能分块策略:")
smart_chunks = smart_chunking(test_data, num_workers)
chunk_sizes = [len(chunk[1]) for chunk in smart_chunks]
print(f"数据块大小: {chunk_sizes}")
with ProcessPoolExecutor(max_workers=num_workers) as executor:
start = time.time()
smart_results = list(executor.map(process_data_chunk, smart_chunks))
smart_time = time.time() - start
print(f"智能分块耗时: {smart_time:.2f}秒")
print(f"性能提升: {naive_time/smart_time:.2f}x")
# 结果汇总
total_naive = sum(result[1] for result in naive_results)
total_smart = sum(result[1] for result in smart_results)
print(f"\n简单分块结果: {total_naive}")
print(f"智能分块结果: {total_smart}")
---
6.3 IO密集型任务优化
01.异步IO优化
a.异步IO原理
异步IO使用非阻塞操作，当遇到IO操作时不会阻塞线程，而是注册回调函数。事件循环在IO完成时调用回调，实现高效并发处理。适合大量IO等待的场景。
b.异步IO示例
---
import asyncio
import aiohttp
import time
import random
from urllib.parse import urljoin
# 模拟异步HTTP请求
async def fetch_url(session, url):
"""异步获取URL内容"""
start_time = time.time()
try:
async with session.get(url, timeout=10) as response:
content = await response.text()
elapsed = time.time() - start_time
print(f"获取 {url} 完成，耗时: {elapsed:.2f}秒，状态: {response.status}")
return {
'url': url,
'status': response.status,
'content_length': len(content),
'time': elapsed
}
except Exception as e:
elapsed = time.time() - start_time
print(f"获取 {url} 失败: {e}，耗时: {elapsed:.2f}秒")
return {
'url': url,
'error': str(e),
'time': elapsed
}
async def async_fetch_all(urls):
"""并发获取所有URL"""
async with aiohttp.ClientSession() as session:
tasks = [fetch_url(session, url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
# 模拟异步文件操作
async def async_file_write(filename, content):
"""异步写入文件"""
start_time = time.time()
await asyncio.sleep(random.uniform(0.1, 0.5)) # 模拟IO延迟
with open(filename, 'w', encoding='utf-8') as f:
f.write(content)
elapsed = time.time() - start_time
print(f"写入 {filename} 完成，耗时: {elapsed:.2f}秒")
return elapsed
async def async_file_read(filename):
"""异步读取文件"""
start_time = time.time()
await asyncio.sleep(random.uniform(0.1, 0.3)) # 模拟IO延迟
with open(filename, 'r', encoding='utf-8') as f:
content = f.read()
elapsed = time.time() - start_time
print(f"读取 {filename} 完成，耗时: {elapsed:.2f}秒，长度: {len(content)}")
return {'filename': filename, 'content': content, 'time': elapsed}
# 测试异步IO
async def main():
print("=== 异步HTTP请求测试 ===")
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/2',
'https://httpbin.org/status/200',
'https://jsonplaceholder.typicode.com/posts/1'
]
start_time = time.time()
results = await async_fetch_all(urls)
total_time = time.time() - start_time
print(f"\n总耗时: {total_time:.2f}秒")
print(f"成功请求数: {len([r for r in results if 'status' in r])}")
print(f"失败请求数: {len([r for r in results if 'error' in r])}")
print("\n=== 异步文件操作测试 ===")
# 创建测试文件
test_files = []
for i in range(5):
filename = f"test_file_{i}.txt"
content = f"这是测试文件 {i} 的内容\n" * 100
test_files.append((filename, content))
# 并发写入文件
start_time = time.time()
write_tasks = [async_file_write(filename, content)
for filename, content in test_files]
write_times = await asyncio.gather(*write_tasks)
write_total = time.time() - start_time
print(f"文件写入总耗时: {write_total:.2f}秒")
print(f"平均写入时间: {sum(write_times)/len(write_times):.2f}秒")
# 并发读取文件
start_time = time.time()
read_tasks = [async_file_read(filename) for filename, _ in test_files]
read_results = await asyncio.gather(*read_tasks)
read_total = time.time() - start_time
print(f"文件读取总耗时: {read_total:.2f}秒")
# 清理测试文件
import os
for filename, _ in test_files:
try:
os.remove(filename)
except:
pass
# 运行异步主函数
if __name__ == "__main__":
asyncio.run(main())
---
02.线程池IO优化
a.线程池原理
使用ThreadPoolExecutor管理线程池，避免频繁创建销毁线程的开销。IO操作期间线程释放GIL，允许其他线程执行。适合同步IO函数的并发化。
b.线程池IO示例
---
import threading
import time
import random
from concurrent.futures import ThreadPoolExecutor, as_completed
import queue
import socket
import requests
# 模拟同步IO操作
def sync_io_task(task_id, duration):
"""同步IO任务"""
start_time = time.time()
time.sleep(duration) # 模拟IO等待
elapsed = time.time() - start_time
print(f"任务 {task_id} 完成，耗时: {elapsed:.2f}秒")
return {'task_id': task_id, 'duration': elapsed}
def network_request(url):
"""网络请求任务"""
start_time = time.time()
try:
response = requests.get(url, timeout=5)
elapsed = time.time() - start_time
print(f"请求 {url} 完成，状态: {response.status_code}，耗时: {elapsed:.2f}秒")
return {
'url': url,
'status': response.status_code,
'content_length': len(response.content),
'time': elapsed
}
except Exception as e:
elapsed = time.time() - start_time
print(f"请求 {url} 失败: {e}，耗时: {elapsed:.2f}秒")
return {'url': url, 'error': str(e), 'time': elapsed}
def file_io_task(filename, operation='write'):
"""文件IO任务"""
start_time = time.time()
try:
if operation == 'write':
content = f"测试内容 {time.time()}\n" * 100
with open(filename, 'w', encoding='utf-8') as f:
f.write(content)
result = {'operation': 'write', 'filename': filename}
else:
with open(filename, 'r', encoding='utf-8') as f:
content = f.read()
result = {'operation': 'read', 'filename': filename, 'length': len(content)}
elapsed = time.time() - start_time
print(f"文件{operation} {filename} 完成，耗时: {elapsed:.2f}秒")
result['time'] = elapsed
return result
except Exception as e:
elapsed = time.time() - start_time
print(f"文件{operation} {filename} 失败: {e}，耗时: {elapsed:.2f}秒")
return {'operation': operation, 'filename': filename, 'error': str(e), 'time': elapsed}
# 使用线程池执行IO任务
def thread_pool_io_demo():
"""线程池IO演示"""
print("=== 线程池IO任务测试 ===")
# 任务1: 模拟IO等待
io_tasks = [(i, random.uniform(0.5, 2.0)) for i in range(6)]
with ThreadPoolExecutor(max_workers=3) as executor:
start_time = time.time()
# 提交所有任务
future_to_task = {
executor.submit(sync_io_task, task_id, duration): (task_id, duration)
for task_id, duration in io_tasks
}
# 收集结果
results = []
for future in as_completed(future_to_task):
task_info = future_to_task[future]
try:
result = future.result()
results.append(result)
except Exception as e:
print(f"任务 {task_info[0]} 异常: {e}")
total_time = time.time() - start_time
print(f"线程池执行总耗时: {total_time:.2f}秒")
print(f"完成任务数: {len(results)}")
print(f"平均任务时间: {sum(r['duration'] for r in results)/len(results):.2f}秒")
# 使用线程池进行网络请求
def thread_pool_network_demo():
"""线程池网络请求演示"""
print("\n=== 线程池网络请求测试 ===")
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/2',
'https://httpbin.org/status/200',
'https://jsonplaceholder.typicode.com/posts/1',
'https://jsonplaceholder.typicode.com/posts/2'
]
with ThreadPoolExecutor(max_workers=3) as executor:
start_time = time.time()
# 提交网络请求任务
future_to_url = {
executor.submit(network_request, url): url
for url in urls
}
# 收集结果
results = []
for future in as_completed(future_to_url):
url = future_to_url[future]
try:
result = future.result()
results.append(result)
except Exception as e:
print(f"请求 {url} 异常: {e}")
total_time = time.time() - start_time
print(f"网络请求总耗时: {total_time:.2f}秒")
print(f"成功请求: {len([r for r in results if 'status' in r])}")
print(f"失败请求: {len([r for r in results if 'error' in r])}")
# 使用线程池进行文件操作
def thread_pool_file_demo():
"""线程池文件操作演示"""
print("\n=== 线程池文件操作测试 ===")
# 创建测试文件名
test_files = [f"pool_test_{i}.txt" for i in range(8)]
with ThreadPoolExecutor(max_workers=4) as executor:
# 写入文件
start_time = time.time()
write_futures = [
executor.submit(file_io_task, filename, 'write')
for filename in test_files
]
write_results = [future.result() for future in as_completed(write_futures)]
write_time = time.time() - start_time
print(f"并发写入耗时: {write_time:.2f}秒")
# 读取文件
start_time = time.time()
read_futures = [
executor.submit(file_io_task, filename, 'read')
for filename in test_files
]
read_results = [future.result() for future in as_completed(read_futures)]
read_time = time.time() - start_time
print(f"并发读取耗时: {read_time:.2f}秒")
# 清理文件
import os
for filename in test_files:
try:
os.remove(filename)
except:
pass
# 比较同步vs异步性能
def performance_comparison():
"""性能对比"""
print("\n=== 性能对比测试 ===")
# 同步执行
def sync_execution():
start_time = time.time()
for i in range(5):
sync_io_task(i, 1.0)
return time.time() - start_time
# 异步执行
def async_execution():
start_time = time.time()
with ThreadPoolExecutor(max_workers=5) as executor:
futures = [executor.submit(sync_io_task, i, 1.0) for i in range(5)]
[f.result() for f in futures]
return time.time() - start_time
sync_time = sync_execution()
async_time = async_execution()
print(f"同步执行耗时: {sync_time:.2f}秒")
print(f"异步执行耗时: {async_time:.2f}秒")
print(f"性能提升: {sync_time/async_time:.2f}x")
if __name__ == "__main__":
thread_pool_io_demo()
thread_pool_network_demo()
thread_pool_file_demo()
performance_comparison()
---
03.IO密集型任务调度策略
a.调度策略原理
根据IO/CPU比率选择合适的并发策略：纯IO密集型使用异步IO，混合型使用线程池，大量IO等待使用协程。合理控制并发数量，避免资源过度竞争。
b.调度策略示例
---
import asyncio
import threading
import time
import random
from concurrent.futures import ThreadPoolExecutor
from typing import List, Dict, Any
import psutil
class IOTaskScheduler:
"""IO任务调度器"""
def __init__(self, max_concurrent=10):
self.max_concurrent = max_concurrent
self.thread_pool = ThreadPoolExecutor(max_workers=max_concurrent//2)
self.semaphore = asyncio.Semaphore(max_concurrent)
self.stats = {
'total_tasks': 0,
'completed_tasks': 0,
'failed_tasks': 0,
'total_time': 0.0
}
async def schedule_async_task(self, task_func, *args, **kwargs):
"""调度异步任务"""
async with self.semaphore:
start_time = time.time()
try:
result = await task_func(*args, **kwargs)
elapsed = time.time() - start_time
self.stats['completed_tasks'] += 1
self.stats['total_time'] += elapsed
return {'result': result, 'time': elapsed, 'type': 'async'}
except Exception as e:
elapsed = time.time() - start_time
self.stats['failed_tasks'] += 1
return {'error': str(e), 'time': elapsed, 'type': 'async'}
finally:
self.stats['total_tasks'] += 1
def schedule_sync_task(self, task_func, *args, **kwargs):
"""调度同步任务"""
start_time = time.time()
try:
result = task_func(*args, **kwargs)
elapsed = time.time() - start_time
self.stats['completed_tasks'] += 1
self.stats['total_time'] += elapsed
return {'result': result, 'time': elapsed, 'type': 'sync'}
except Exception as e:
elapsed = time.time() - start_time
self.stats['failed_tasks'] += 1
return {'error': str(e), 'time': elapsed, 'type': 'sync'}
finally:
self.stats['total_tasks'] += 1
async def schedule_thread_task(self, task_func, *args, **kwargs):
"""调度线程池任务"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
self.thread_pool,
lambda: self.schedule_sync_task(task_func, *args, **kwargs)
)
def get_stats(self):
"""获取统计信息"""
if self.stats['completed_tasks'] > 0:
avg_time = self.stats['total_time'] / self.stats['completed_tasks']
else:
avg_time = 0
return {
**self.stats,
'average_time': avg_time,
'success_rate': self.stats['completed_tasks'] / max(self.stats['total_tasks'], 1)
}
def close(self):
"""关闭调度器"""
self.thread_pool.shutdown()
# 示例任务函数
async def async_io_task(task_id, io_time):
"""异步IO任务"""
await asyncio.sleep(io_time)
return {'task_id': task_id, 'message': f'异步任务 {task_id} 完成'}
def sync_io_task(task_id, io_time):
"""同步IO任务"""
time.sleep(io_time)
return {'task_id': task_id, 'message': f'同步任务 {task_id} 完成'}
def cpu_intensive_task(task_id, cpu_time):
"""CPU密集型任务"""
count = 0
end_time = time.time() + cpu_time
while time.time() < end_time:
count += 1
return {'task_id': task_id, 'count': count}
# 混合任务负载测试
async def mixed_workload_test():
"""混合工作负载测试"""
print("=== 混合工作负载测试 ===")
scheduler = IOTaskScheduler(max_concurrent=8)
# 创建不同类型的任务
tasks = []
# 异步IO任务
for i in range(5):
tasks.append(
scheduler.schedule_async_task(async_io_task, f"async-{i}", random.uniform(0.5, 1.5))
)
# 同步IO任务（使用线程池）
for i in range(3):
tasks.append(
scheduler.schedule_thread_task(sync_io_task, f"sync-{i}", random.uniform(0.3, 1.0))
)
# CPU密集型任务（使用线程池）
for i in range(2):
tasks.append(
scheduler.schedule_thread_task(cpu_intensive_task, f"cpu-{i}", random.uniform(0.2, 0.5))
)
# 执行所有任务
start_time = time.time()
results = await asyncio.gather(*tasks, return_exceptions=True)
total_time = time.time() - start_time
# 分析结果
async_results = [r for r in results if isinstance(r, dict) and r.get('type') == 'async']
sync_results = [r for r in results if isinstance(r, dict) and r.get('type') == 'sync']
print(f"总执行时间: {total_time:.2f}秒")
print(f"异步任务: {len(async_results)}个")
print(f"同步任务: {len(sync_results)}个")
# 获取调度器统计
stats = scheduler.get_stats()
print(f"调度器统计:")
print(f" 总任务数: {stats['total_tasks']}")
print(f" 成功任务: {stats['completed_tasks']}")
print(f" 失败任务: {stats['failed_tasks']}")
print(f" 平均耗时: {stats['average_time']:.2f}秒")
print(f" 成功率: {stats['success_rate']:.2%}")
scheduler.close()
# 动态并发控制
async def dynamic_concurrency_test():
"""动态并发控制测试"""
print("\n=== 动态并发控制测试 ===")
# 获取系统信息
cpu_count = psutil.cpu_count()
memory_gb = psutil.virtual_memory().total / (1024**3)
print(f"系统CPU核心数: {cpu_count}")
print(f"系统内存: {memory_gb:.1f}GB")
# 根据系统资源动态调整并发数
optimal_concurrent = min(cpu_count * 3, 20) # 基于CPU核心数
print(f"推荐并发数: {optimal_concurrent}")
scheduler = IOTaskScheduler(max_concurrent=optimal_concurrent)
# 创建大量IO任务
task_count = 50
tasks = []
for i in range(task_count):
if i % 3 == 0:
# 异步任务
tasks.append(
scheduler.schedule_async_task(async_io_task, f"async-{i}", random.uniform(0.1, 0.5))
)
else:
# 同步任务
tasks.append(
scheduler.schedule_thread_task(sync_io_task, f"sync-{i}", random.uniform(0.1, 0.3))
)
# 监控执行过程
start_time = time.time()
results = await asyncio.gather(*tasks, return_exceptions=True)
total_time = time.time() - start_time
print(f"\n执行 {task_count} 个任务:")
print(f"总耗时: {total_time:.2f}秒")
print(f"吞吐量: {task_count/total_time:.1f} 任务/秒")
stats = scheduler.get_stats()
print(f"平均任务时间: {stats['average_time']:.2f}秒")
print(f"并发效率: {stats['average_time'] * optimal_concurrent / total_time:.2%}")
scheduler.close()
# IO密集型负载基准测试
async def io_benchmark():
"""IO密集型基准测试"""
print("\n=== IO密集型基准测试 ===")
for concurrent in [2, 4, 8, 16]:
print(f"\n测试并发数: {concurrent}")
scheduler = IOTaskScheduler(max_concurrent=concurrent)
# 创建IO任务
task_count = concurrent * 4
tasks = [
scheduler.schedule_async_task(async_io_task, f"task-{i}", random.uniform(0.1, 0.2))
for i in range(task_count)
]
# 执行并计时
start_time = time.time()
results = await asyncio.gather(*tasks, return_exceptions=True)
exec_time = time.time() - start_time
stats = scheduler.get_stats()
print(f" 执行时间: {exec_time:.2f}秒")
print(f" 吞吐量: {task_count/exec_time:.1f} 任务/秒")
print(f" 平均延迟: {stats['average_time']:.2f}秒")
print(f" 成功率: {stats['success_rate']:.2%}")
scheduler.close()
# 短暂休息避免资源竞争
await asyncio.sleep(1)
if __name__ == "__main__":
asyncio.run(mixed_workload_test())
asyncio.run(dynamic_concurrency_test())
asyncio.run(io_benchmark())
---
6.4 多进程绕过GIL
01.multiprocessing基础
a.多进程原理
multiprocessing创建独立进程，每个进程有独立的GIL，实现真正的并行执行。进程间通信使用IPC机制，包括管道、队列、共享内存等。适合CPU密集型任务的并行化。
b.multiprocessing基础示例
---
import multiprocessing
import time
import os
import random
from multiprocessing import Process, Queue, Pipe, Manager
import numpy as np
def cpu_intensive_task(n):
"""CPU密集型任务"""
pid = os.getpid()
print(f"进程 {pid} 开始计算，参数: {n}")
start_time = time.time()
result = sum(i * i for i in range(n))
elapsed = time.time() - start_time
print(f"进程 {pid} 完成，结果: {result:,}，耗时: {elapsed:.2f}秒")
return result
def process_worker(task_id, n, result_queue):
"""进程工作函数"""
try:
result = cpu_intensive_task(n)
result_queue.put({'task_id': task_id, 'pid': os.getpid(),
'result': result, 'status': 'success'})
except Exception as e:
result_queue.put({'task_id': task_id, 'pid': os.getpid(),
'error': str(e), 'status': 'error'})
# 基础多进程测试
def basic_multiprocessing_test():
"""基础多进程测试"""
print("=== 基础多进程测试 ===")
tasks = [(i, 100000 + i * 50000) for i in range(4)]
result_queue = multiprocessing.Queue()
# 创建进程
processes = []
for task_id, n in tasks:
p = Process(target=process_worker, args=(task_id, n, result_queue))
processes.append(p)
p.start()
print(f"启动进程 {p.pid} 执行任务 {task_id}")
# 等待所有进程完成
for p in processes:
p.join()
print(f"进程 {p.pid} 已结束")
# 收集结果
results = []
while not result_queue.empty():
results.append(result_queue.get())
print(f"\n完成 {len(results)} 个任务")
for result in results:
if result['status'] == 'success':
print(f"任务 {result['task_id']}: {result['result']:,}")
else:
print(f"任务 {result['task_id']}: 失败 - {result['error']}")
# 进程池测试
def process_pool_test():
"""进程池测试"""
print("\n=== 进程池测试 ===")
# 单进程测试
print("单进程执行:")
start_time = time.time()
single_result = cpu_intensive_task(500000)
single_time = time.time() - start_time
print(f"单进程耗时: {single_time:.2f}秒，结果: {single_result:,}")
# 多进程测试
print("\n多进程执行:")
with multiprocessing.Pool(processes=4) as pool:
start_time = time.time()
tasks = [500000] * 4
results = pool.map(cpu_intensive_task, tasks)
multi_time = time.time() - start_time
print(f"多进程耗时: {multi_time:.2f}秒")
print(f"性能提升: {single_time/multi_time:.2f}x")
print(f"结果: {[f'{r:,}' for r in results]}")
# 进程ID和内存信息
def process_info_test():
"""进程信息测试"""
print("\n=== 进程信息测试 ===")
def print_process_info(name):
"""打印进程信息"""
pid = os.getpid()
ppid = os.getppid()
print(f"{name}: PID={pid}, PPID={ppid}")
# 打印内存信息（如果可用）
try:
import psutil
process = psutil.Process(pid)
memory_mb = process.memory_info().rss / 1024 / 1024
print(f"内存使用: {memory_mb:.1f}MB")
except ImportError:
print("psutil模块不可用，无法显示内存信息")
print_process_info("主进程")
with multiprocessing.Pool(processes=2) as pool:
# 在子进程中打印信息
pool.map(print_process_info, ["子进程1", "子进程2"])
if __name__ == "__main__":
# 注意：multiprocessing需要在主模块中执行
basic_multiprocessing_test()
process_pool_test()
process_info_test()
---
02.进程间通信
a.IPC机制原理
进程间不能直接共享内存，需要使用IPC机制。Queue提供线程安全的队列通信，Pipe建立双向通道，Manager支持共享数据结构，Array/Value支持共享内存。
b.IPC通信示例
---
import multiprocessing
import time
import random
from multiprocessing import Process, Queue, Pipe, Manager, Array, Value
import json
def producer_worker(task_queue, result_queue, num_items):
"""生产者进程"""
pid = multiprocessing.current_process().pid
print(f"生产者进程 {pid} 开始生产 {num_items} 个项目")
for i in range(num_items):
item = {
'id': i,
'data': f"数据_{i}",
'timestamp': time.time(),
'producer': pid
}
task_queue.put(item)
time.sleep(random.uniform(0.1, 0.3)) # 模拟生产时间
print(f"生产者进程 {pid} 完成")
def consumer_worker(task_queue, result_queue, num_items):
"""消费者进程"""
pid = multiprocessing.current_process().pid
print(f"消费者进程 {pid} 开始消费")
consumed = 0
while consumed < num_items:
try:
item = task_queue.get(timeout=1)
# 模拟处理时间
time.sleep(random.uniform(0.2, 0.5))
result = {
'original_id': item['id'],
'processed_data': f"已处理_{item['data']}",
'consumer': pid,
'process_time': time.time()
}
result_queue.put(result)
consumed += 1
print(f"消费者 {pid} 处理了项目 {item['id']}")
except Exception:
if consumed > 0:
break
continue
print(f"消费者进程 {pid} 完成，处理了 {consumed} 个项目")
# Queue通信测试
def queue_communication_test():
"""Queue通信测试"""
print("=== Queue通信测试 ===")
task_queue = multiprocessing.Queue()
result_queue = multiprocessing.Queue()
num_items = 10
# 启动生产者和消费者
producer = Process(target=producer_worker,
args=(task_queue, result_queue, num_items))
consumer = Process(target=consumer_worker,
args=(task_queue, result_queue, num_items))
start_time = time.time()
producer.start()
consumer.start()
producer.join()
consumer.join()
total_time = time.time() - start_time
# 收集结果
results = []
while not result_queue.empty():
results.append(result_queue.get())
print(f"\n总耗时: {total_time:.2f}秒")
print(f"处理了 {len(results)} 个项目")
print(f"平均处理时间: {total_time/len(results):.2f}秒/项")
# Pipe通信测试
def pipe_communication_test():
"""Pipe通信测试"""
print("\n=== Pipe通信测试 ===")
def ping_pong_worker(conn, name):
"""Ping-Pong测试"""
for i in range(5):
# 接收消息
message = conn.recv()
print(f"{name} 接收到: {message}")
# 发送响应
response = f"{name} 响应 #{i+1}"
conn.send(response)
time.sleep(0.2)
conn.close()
# 创建管道
parent_conn, child_conn = Pipe()
# 启动子进程
child = Process(target=ping_pong_worker, args=(child_conn, "子进程"))
child.start()
# 父进程发送消息
for i in range(5):
message = f"父进程消息 #{i+1}"
parent_conn.send(message)
response = parent_conn.recv()
print(f"父进程接收到响应: {response}")
time.sleep(0.1)
child.join()
# Manager共享数据测试
def manager_shared_data_test():
"""Manager共享数据测试"""
print("\n=== Manager共享数据测试 ===")
def shared_dict_worker(shared_dict, key_prefix, num_items):
"""共享字典工作进程"""
pid = multiprocessing.current_process().pid
for i in range(num_items):
key = f"{key_prefix}_{i}"
value = {
'data': f"{key}_value",
'pid': pid,
'timestamp': time.time()
}
shared_dict[key] = value
time.sleep(0.1)
print(f"进程 {pid} 添加了 {num_items} 个项目")
def shared_list_worker(shared_list, start_idx, num_items):
"""共享列表工作进程"""
pid = multiprocessing.current_process().pid
for i in range(num_items):
item = {
'index': start_idx + i,
'pid': pid,
'data': f"item_{start_idx + i}"
}
shared_list.append(item)
time.sleep(0.05)
print(f"进程 {pid} 添加了 {num_items} 个项目")
# 使用Manager创建共享对象
with Manager() as manager:
shared_dict = manager.dict()
shared_list = manager.list()
# 启动多个工作进程
processes = []
# 字典工作进程
for i in range(2):
p = Process(target=shared_dict_worker,
args=(shared_dict, f"dict_part_{i}", 5))
processes.append(p)
p.start()
# 列表工作进程
for i in range(2):
p = Process(target=shared_list_worker,
args=(shared_list, i * 5, 5))
processes.append(p)
p.start()
# 等待所有进程完成
for p in processes:
p.join()
# 检查共享数据
print(f"\n共享字典包含 {len(shared_dict)} 个项目:")
for key, value in list(shared_dict.items())[:5]:
print(f" {key}: {value}")
print(f"\n共享列表包含 {len(shared_list)} 个项目:")
for item in shared_list[:5]:
print(f" {item}")
# Array/Value共享内存测试
def shared_memory_test():
"""共享内存测试"""
print("\n=== 共享内存测试 ===")
def array_worker(shared_array, start_idx, length, multiplier):
"""数组工作进程"""
pid = multiprocessing.current_process().pid
for i in range(length):
shared_array[start_idx + i] = (start_idx + i) * multiplier
time.sleep(0.05)
print(f"进程 {pid} 处理了数组索引 {start_idx}-{start_idx+length-1}")
def counter_worker(counter):
"""计数器工作进程"""
pid = multiprocessing.current_process().pid
for _ in range(1000):
with counter.get_lock():
counter.value += 1
time.sleep(0.001) # 模拟处理
print(f"进程 {pid} 完成计数，当前值: {counter.value}")
# 创建共享数组和值
shared_array = Array('i', 20) # 整数数组，20个元素
counter = Value('i', 0) # 共享计数器
# 启动数组工作进程
array_processes = []
for i in range(4):
p = Process(target=array_worker,
args=(shared_array, i * 5, 5, (i + 1)))
array_processes.append(p)
p.start()
# 启动计数器进程
counter_processes = []
for _ in range(5):
p = Process(target=counter_worker, args=(counter,))
counter_processes.append(p)
p.start()
# 等待所有进程完成
for p in array_processes + counter_processes:
p.join()
# 检查结果
print(f"\n共享数组内容:")
for i in range(len(shared_array)):
print(f" [{i}]: {shared_array[i]}")
print(f"\n最终计数器值: {counter.value}")
if __name__ == "__main__":
queue_communication_test()
pipe_communication_test()
manager_shared_data_test()
shared_memory_test()
---
03.进程池优化
a.进程池原理
进程池预先创建固定数量的工作进程，避免进程创建销毁开销。支持任务队列管理、超时控制、回调函数等高级功能。可以根据CPU核心数和任务特性优化池大小。
b.进程池优化示例
---
import multiprocessing
import time
import random
from multiprocessing import Pool, cpu_count
from functools import partial
import numpy as np
from concurrent.futures import ProcessPoolExecutor
class OptimizedProcessPool:
"""优化的进程池实现"""
def __init__(self, processes=None, timeout=30):
self.processes = processes or cpu_count()
self.timeout = timeout
self.stats = {
'total_tasks': 0,
'completed_tasks': 0,
'failed_tasks': 0,
'total_time': 0.0
}
def optimize_chunk_size(self, data_size, chunk_size=None):
"""优化块大小"""
if chunk_size is None:
# 根据数据大小和进程数计算最佳块大小
base_chunk = max(1, data_size // (self.processes * 4))
chunk_size = min(base_chunk, 1000)
return chunk_size
def chunked_processing(self, data, worker_func, chunk_size=None):
"""分块处理数据"""
chunk_size = self.optimize_chunk_size(len(data), chunk_size)
chunks = [data[i:i + chunk_size] for i in range(0, len(data), chunk_size)]
print(f"数据分为 {len(chunks)} 块，每块大小约 {chunk_size}")
return chunks
def matrix_multiply_worker(matrix_a_chunk, matrix_b):
"""矩阵乘法工作函数"""
"""接收矩阵A的一个块和完整的矩阵B，计算部分结果"""
result_chunk = []
for row in matrix_a_chunk:
# 计算一行与矩阵B的乘积
result_row = []
for col in range(len(matrix_b[0])):
# 计算矩阵行与列的点积
dot_product = sum(row[i] * matrix_b[i][col] for i in range(len(row)))
result_row.append(dot_product)
result_chunk.append(result_row)
return result_chunk
# 矩阵乘法并行计算
def parallel_matrix_multiplication():
"""并行矩阵乘法"""
print("=== 并行矩阵乘法测试 ===")
# 创建测试矩阵
size = 200
print(f"创建 {size}x{size} 的矩阵...")
matrix_a = [[random.randint(1, 10) for _ in range(size)] for _ in range(size)]
matrix_b = [[random.randint(1, 10) for _ in range(size)] for _ in range(size)]
# 单进程计算
print("单进程计算矩阵乘法...")
start_time = time.time()
result_single = []
for i in range(size):
row = []
for j in range(size):
dot_product = sum(matrix_a[i][k] * matrix_b[k][j] for k in range(size))
row.append(dot_product)
result_single.append(row)
single_time = time.time() - start_time
print(f"单进程耗时: {single_time:.2f}秒")
# 多进程计算
print(f"\n多进程计算矩阵乘法（{cpu_count()}个进程）...")
pool = OptimizedProcessPool(processes=cpu_count())
# 将矩阵A按行分块
chunks = pool.chunked_processing(matrix_a)
start_time = time.time()
with ProcessPoolExecutor(max_workers=cpu_count()) as executor:
# 提交矩阵乘法任务
futures = [
executor.submit(matrix_multiply_worker, chunk, matrix_b)
for chunk in chunks
]
# 收集结果
result_chunks = [future.result() for future in futures]
# 合并结果块
result_parallel = []
for chunk in result_chunks:
result_parallel.extend(chunk)
parallel_time = time.time() - start_time
# 验证结果
if len(result_single) == len(result_parallel):
first_diff = False
for i in range(min(5, len(result_single))):
for j in range(min(5, len(result_single[0]))):
if result_single[i][j] != result_parallel[i][j]:
first_diff = True
break
if first_diff:
break
if not first_diff:
print("✓ 并行计算结果正确")
else:
print("✗ 并行计算结果有误")
print(f"多进程耗时: {parallel_time:.2f}秒")
print(f"性能提升: {single_time/parallel_time:.2f}x")
def image_processing_worker(image_data, operation):
"""图像处理工作函数"""
"""模拟图像处理操作"""
height, width = len(image_data), len(image_data[0])
result = [[0] * width for _ in range(height)]
if operation == 'blur':
# 模拟模糊处理
for i in range(1, height-1):
for j in range(1, width-1):
# 简单的平均值模糊
total = 0
count = 0
for di in [-1, 0, 1]:
for dj in [-1, 0, 1]:
ni, nj = i + di, j + dj
if 0 <= ni < height and 0 <= nj < width:
total += image_data[ni][nj]
count += 1
result[i][j] = total // count
elif operation == 'edge':
# 模拟边缘检测
for i in range(1, height-1):
for j in range(1, width-1):
# 简单的梯度计算
gx = image_data[i][j+1] - image_data[i][j-1]
gy = image_data[i+1][j] - image_data[i-1][j]
result[i][j] = min(255, abs(gx) + abs(gy))
else:
# 原样复制
for i in range(height):
for j in range(width):
result[i][j] = image_data[i][j]
return result
# 并行图像处理
def parallel_image_processing():
"""并行图像处理"""
print("\n=== 并行图像处理测试 ===")
# 创建测试图像数据
height, width = 500, 500
print(f"创建 {height}x{width} 的测试图像...")
# 创建随机图像数据
image_data = [[random.randint(0, 255) for _ in range(width)] for _ in range(height)]
operations = ['blur', 'edge', 'copy']
for operation in operations:
print(f"\n处理操作: {operation}")
# 单进程处理
start_time = time.time()
result_single = image_processing_worker(image_data, operation)
single_time = time.time() - start_time
print(f"单进程耗时: {single_time:.2f}秒")
# 多进程处理
pool = OptimizedProcessPool()
# 将图像按行分块
chunk_size = height // cpu_count()
chunks = [
image_data[i:i + chunk_size]
for i in range(0, height, chunk_size)
]
start_time = time.time()
with ProcessPoolExecutor(max_workers=cpu_count()) as executor:
futures = [
executor.submit(image_processing_worker, chunk, operation)
for chunk in chunks
]
result_chunks = [future.result() for future in futures]
# 合并结果
result_parallel = []
for chunk in result_chunks:
result_parallel.extend(chunk)
parallel_time = time.time() - start_time
print(f"多进程耗时: {parallel_time:.2f}秒")
print(f"性能提升: {single_time/parallel_time:.2f}x")
def data_analysis_worker(data_chunk, analysis_type):
"""数据分析工作函数"""
"""模拟大数据分析任务"""
if analysis_type == 'statistical':
# 统计分析
data_flat = [item for row in data_chunk for item in row]
return {
'mean': sum(data_flat) / len(data_flat),
'max': max(data_flat),
'min': min(data_flat),
'count': len(data_flat)
}
elif analysis_type == 'aggregation':
# 聚合分析
row_sums = [sum(row) for row in data_chunk]
return {
'row_sum_total': sum(row_sums),
'row_sum_avg': sum(row_sums) / len(row_sums),
'row_count': len(data_chunk)
}
else:
# 自定义分析
total = sum(sum(row) for row in data_chunk)
avg = total / (len(data_chunk) * len(data_chunk[0]) if data_chunk else 1)
return {
'total': total,
'average': avg,
'rows': len(data_chunk)
}
# 并行数据分析
def parallel_data_analysis():
"""并行数据分析"""
print("\n=== 并行数据分析测试 ===")
# 创建大型数据集
rows, cols = 1000, 100
print(f"创建 {rows}x{cols} 的数据集...")
data = [[random.randint(1, 1000) for _ in range(cols)] for _ in range(rows)]
analysis_types = ['statistical', 'aggregation', 'custom']
for analysis_type in analysis_types:
print(f"\n分析类型: {analysis_type}")
# 单进程分析
start_time = time.time()
result_single = data_analysis_worker(data, analysis_type)
single_time = time.time() - start_time
print(f"单进程耗时: {single_time:.2f}秒")
# 多进程分析
pool = OptimizedProcessPool()
# 数据分块
chunk_size = rows // cpu_count()
chunks = [
data[i:i + chunk_size]
for i in range(0, rows, chunk_size)
]
start_time = time.time()
with ProcessPoolExecutor(max_workers=cpu_count()) as executor:
futures = [
executor.submit(data_analysis_worker, chunk, analysis_type)
for chunk in chunks
]
results = [future.result() for future in futures]
# 合并结果
if analysis_type == 'statistical':
all_means = [r['mean'] for r in results]
all_data_flat = sum(r['count'] for r in results)
merged_result = {
'mean': sum(r['mean'] * r['count'] for r in results) / all_data_flat,
'max': max(r['max'] for r in results),
'min': min(r['min'] for r in results),
'count': all_data_flat
}
elif analysis_type == 'aggregation':
merged_result = {
'row_sum_total': sum(r['row_sum_total'] for r in results),
'row_sum_avg': sum(r['row_sum_avg'] for r in results) / len(results),
'row_count': sum(r['row_count'] for r in results)
}
else:
merged_result = {
'total': sum(r['total'] for r in results),
'average': sum(r['average'] for r in results) / len(results),
'rows': sum(r['rows'] for r in results)
}
parallel_time = time.time() - start_time
print(f"多进程耗时: {parallel_time:.2f}秒")
print(f"性能提升: {single_time/parallel_time:.2f}x")
if __name__ == "__main__":
print(f"CPU核心数: {cpu_count()}")
parallel_matrix_multiplication()
parallel_image_processing()
parallel_data_analysis()
---
6.5 性能分析工具
01.cProfile性能分析
a.cProfile原理
cProfile是Python内置的性能分析工具，可以统计函数调用次数、执行时间、调用关系等信息。通过分析可以识别性能瓶颈，优化代码结构。支持文本和图形化输出格式。
b.cProfile性能分析示例
---
import cProfile
import pstats
import time
import threading
import multiprocessing
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import io
# 性能测试函数
def fibonacci(n):
"""计算斐波那契数列（递归实现，用于性能测试）"""
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
def bubble_sort(arr):
"""冒泡排序算法"""
n = len(arr)
for i in range(n):
for j in range(0, n-i-1):
if arr[j] > arr[j+1]:
arr[j], arr[j+1] = arr[j+1], arr[j]
return arr
def matrix_multiplication(size):
"""矩阵乘法"""
# 创建两个矩阵
matrix_a = [[i for i in range(size)] for _ in range(size)]
matrix_b = [[j for j in range(size)] for _ in range(size)]
# 矩阵乘法
result = [[0] * size for _ in range(size)]
for i in range(size):
for j in range(size):
for k in range(size):
result[i][j] += matrix_a[i][k] * matrix_b[k][j]
return result
def io_simulation():
"""模拟IO操作"""
time.sleep(0.1) # 模拟IO等待
return "IO operation completed"
def cpu_intensive_task():
"""CPU密集型任务"""
total = 0
for i in range(1000000):
total += i * i
return total
# 测试函数组合
def performance_test_suite():
"""性能测试套件"""
print("执行性能测试套件...")
# 斐波那契计算
fib_result = fibonacci(20)
print(f"斐波那契结果: {fib_result}")
# 排序测试
test_array = list(range(100, 0, -1))
sorted_array = bubble_sort(test_array.copy())
print(f"排序完成，数组长度: {len(sorted_array)}")
# 矩阵乘法
matrix_result = matrix_multiplication(50)
print(f"矩阵乘法完成，结果维度: {len(matrix_result)}x{len(matrix_result[0])}")
# IO操作
io_result = io_simulation()
print(f"IO操作结果: {io_result}")
# CPU密集型任务
cpu_result = cpu_intensive_task()
print(f"CPU任务完成，总和: {cpu_result}")
# cProfile基本使用
def basic_cprofile_demo():
"""cProfile基本演示"""
print("=== cProfile基本性能分析 ===")
# 创建性能分析器
profiler = cProfile.Profile()
# 开始分析
profiler.enable()
# 执行被分析的代码
performance_test_suite()
# 停止分析
profiler.disable()
# 输出统计信息
print("\n=== 性能统计信息 ===")
stats = pstats.Stats(profiler)
stats.sort_stats('cumulative') # 按累计时间排序
# 打印前20个函数
print("累计时间排序的前20个函数:")
stats.print_stats(20)
# 按函数自身时间排序
print("\n函数自身时间排序的前20个函数:")
stats.sort_stats('time')
stats.print_stats(20)
# cProfile命令行使用
def command_line_cprofile():
"""命令行cProfile使用"""
print("\n=== 命令行cProfile使用 ===")
print("可以通过以下命令运行性能分析:")
print("python -m cProfile -s cumulative script.py")
print("python -m cProfile -o profile_output.prof script.py")
# 保存和分析性能数据
def save_and_analyze_profile():
"""保存和分析性能数据"""
print("\n=== 保存和分析性能数据 ===")
# 创建分析器并执行
profiler = cProfile.Profile()
profiler.enable()
# 执行多次测试以获得更准确的数据
for i in range(3):
print(f"执行测试 {i+1}/3")
performance_test_suite()
profiler.disable()
# 保存性能数据到文件
profiler.dump_stats("performance_profile.prof")
print("性能数据已保存到 performance_profile.prof")
# 分析保存的数据
print("\n分析保存的性能数据:")
stats = pstats.Stats("performance_profile.prof")
# 按不同方式排序
print("\n按累计时间排序:")
stats.sort_stats('cumulative')
stats.print_stats(10)
print("\n按调用次数排序:")
stats.sort_stats('calls')
stats.print_stats(10)
# 生成详细报告
print("\n生成详细报告到文件...")
with open("performance_report.txt", "w") as f:
sys.stdout = f
stats.print_stats()
sys.stdout = sys.__stdout__
print("详细报告已保存到 performance_report.txt")
# 函数调用关系分析
def call_graph_analysis():
"""函数调用关系分析"""
print("\n=== 函数调用关系分析 ===")
profiler = cProfile.Profile()
profiler.enable()
# 执行特定函数以分析调用关系
bubble_sort(list(range(50, 0, -1)))
profiler.disable()
stats = pstats.Stats(profiler)
stats.sort_stats('cumulative')
# 打印调用者信息
print("bubble_sort函数的调用者:")
stats.print_callers('bubble_sort')
# 打印被调用函数信息
print("\nbubble_sort函数调用的函数:")
stats.print_callees('bubble_sort')
# 性能比较分析
def performance_comparison():
"""性能比较分析"""
print("\n=== 性能比较分析 ===")
# 不同的排序实现
def optimized_bubble_sort(arr):
"""优化的冒泡排序"""
n = len(arr)
for i in range(n):
swapped = False
for j in range(0, n-i-1):
if arr[j] > arr[j+1]:
arr[j], arr[j+1] = arr[j+1], arr[j]
swapped = True
if not swapped:
break
return arr
def quick_sort(arr):
"""快速排序"""
if len(arr) <= 1:
return arr
pivot = arr[len(arr) // 2]
left = [x for x in arr if x < pivot]
middle = [x for x in arr if x == pivot]
right = [x for x in arr if x > pivot]
return quick_sort(left) + middle + quick_sort(right)
# 测试数据
test_data = list(range(100, 0, -1))
# 分析冒泡排序
print("分析冒泡排序:")
profiler1 = cProfile.Profile()
profiler1.enable()
bubble_sort(test_data.copy())
profiler1.disable()
# 分析快速排序
print("分析快速排序:")
profiler2 = cProfile.Profile()
profiler2.enable()
quick_sort(test_data.copy())
profiler2.disable()
# 比较结果
stats1 = pstats.Stats(profiler1, stream=io.StringIO())
stats1.sort_stats('time')
stats1.print_stats(5)
stats2 = pstats.Stats(profiler2, stream=io.StringIO())
stats2.sort_stats('time')
stats2.print_stats(5)
if __name__ == "__main__":
import sys
basic_cprofile_demo()
command_line_cprofile()
save_and_analyze_profile()
call_graph_analysis()
performance_comparison()
---
02.line_profiler行级分析
a.line_profiler原理
line_profiler提供逐行级别的性能分析，可以显示每行代码的执行时间和执行次数。对于优化算法中的关键代码段特别有用。需要使用@profile装饰器标记要分析的函数。
b.line_profiler行级分析示例
---
import time
import random
import numpy as np
from functools import lru_cache
# 需要使用line_profiler分析的函数
def complex_calculation(n):
"""复杂计算函数"""
total = 0
# 第一部分：简单循环
for i in range(n):
total += i * i
# 第二部分：条件判断密集型操作
for i in range(n):
if i % 2 == 0:
total += i
elif i % 3 == 0:
total += i * 2
else:
total -= i
# 第三部分：列表操作
data = []
for i in range(n):
data.append(i * 3)
# 第四部分：字符串操作
result = ""
for i in range(min(100, n)):
result += str(data[i])
return total, len(result)
def matrix_operations(size):
"""矩阵操作函数"""
# 创建矩阵
matrix1 = [[random.random() for _ in range(size)] for _ in range(size)]
matrix2 = [[random.random() for _ in range(size)] for _ in range(size)]
# 矩阵加法
result_add = []
for i in range(size):
row = []
for j in range(size):
row.append(matrix1[i][j] + matrix2[i][j])
result_add.append(row)
# 矩阵乘法（优化版本）
result_mul = [[0] * size for _ in range(size)]
for i in range(size):
for k in range(size):
for j in range(size):
result_mul[i][j] += matrix1[i][k] * matrix2[k][j]
return result_add, result_mul
def data_processing_pipeline(data):
"""数据处理管道"""
processed = []
for item in data:
# 数据清洗
if item > 0:
cleaned = item
else:
cleaned = 0
# 数据转换
transformed = cleaned * 2.5
# 数据验证
if transformed > 1000:
validated = 1000
else:
validated = transformed
processed.append(validated)
# 聚合操作
total = sum(processed)
average = total / len(processed) if processed else 0
# 统计计算
max_value = max(processed) if processed else 0
min_value = min(processed) if processed else 0
return {
'total': total,
'average': average,
'max': max_value,
'min': min_value,
'count': len(processed)
}
def string_processing(text_list):
"""字符串处理函数"""
results = []
for text in text_list:
# 基本处理
cleaned = text.strip().lower()
# 字符串操作
words = cleaned.split()
word_count = len(words)
# 字符统计
char_count = len(cleaned)
space_count = cleaned.count(' ')
# 高级处理
reversed_text = cleaned[::-1]
palindromic = cleaned == reversed_text
# 存储结果
results.append({
'original': text,
'cleaned': cleaned,
'word_count': word_count,
'char_count': char_count,
'space_count': space_count,
'reversed': reversed_text,
'palindromic': palindromic
})
# 聚合统计
total_words = sum(r['word_count'] for r in results)
total_chars = sum(r['char_count'] for r in results)
palindromic_count = sum(1 for r in results if r['palindromic'])
return results, {
'total_words': total_words,
'total_chars': total_chars,
'palindromic_count': palindromic_count,
'processed_count': len(results)
}
# 生成测试数据的辅助函数
def generate_test_data():
"""生成测试数据"""
# 数字数据
numbers = [random.randint(-100, 2000) for _ in range(1000)]
# 字符串数据
texts = [
"Hello World",
"Python Programming",
"Performance Analysis",
"Code Optimization",
"Line Profiler",
"Memory Management",
"Algorithm Design",
"Data Structures",
"Software Engineering",
"System Architecture"
] * 100
return numbers, texts
# 模拟line_profiler的分析结果
def simulate_line_profiler_analysis():
"""模拟line_profiler分析"""
print("=== Line Profiler模拟分析 ===")
print("注意：实际使用需要安装line_profiler并在函数前添加@profile装饰器")
print("安装命令: pip install line_profiler")
print("使用命令: kernprof -l -v script.py\n")
# 生成测试数据
numbers, texts = generate_test_data()
# 执行被分析的函数
print("执行复杂计算函数...")
calc_result = complex_calculation(10000)
print(f"计算结果: 总和={calc_result[0]}, 字符串长度={calc_result[1]}")
print("\n执行矩阵操作函数...")
matrix_result = matrix_operations(50)
print(f"矩阵操作完成: 加法结果{len(matrix_result[0])}x{len(matrix_result[0][0])}")
print("\n执行数据处理管道...")
data_result = data_processing_pipeline(numbers)
print(f"数据处理完成: 平均值={data_result['average']:.2f}")
print("\n执行字符串处理函数...")
string_result, string_stats = string_processing(texts)
print(f"字符串处理完成: 总单词数={string_stats['total_words']}")
# 性能优化建议
def performance_optimization_tips():
"""性能优化建议"""
print("\n=== 基于Line Profiler的优化建议 ===")
optimization_tips = [
{
'category': '循环优化',
'tips': [
'避免在循环中进行重复计算',
'使用列表推导式替代显式循环',
'考虑使用内置函数如map()、filter()',
'预计算循环边界值'
]
},
{
'category': '数据结构优化',
'tips': [
'选择合适的数据结构（集合 vs 列表）',
'使用生成器减少内存使用',
'避免不必要的数据复制',
'考虑使用numpy数组进行数值计算'
]
},
{
'category': '算法优化',
'tips': [
'选择时间复杂度更低的算法',
'使用缓存（如lru_cache）避免重复计算',
'分治策略处理大数据集',
'空间换时间优化'
]
},
{
'category': 'Python特有优化',
'tips': [
'使用局部变量替代全局变量',
'避免在循环中使用try-except',
'使用内置方法和函数',
'考虑使用C扩展或Cython'
]
}
]
for category_info in optimization_tips:
print(f"\n{category_info['category']}:")
for tip in category_info['tips']:
print(f" • {tip}")
# 实际优化示例
def optimization_example():
"""优化示例"""
print("\n=== 代码优化示例 ===")
# 未优化的版本
def unoptimized_function(data):
"""未优化的函数"""
result = []
for i in range(len(data)):
for j in range(len(data)):
if i != j and data[i] == data[j]:
duplicate = True
break
if not duplicate:
result.append(data[i])
return result
# 优化版本1：使用集合
def optimized_v1_function(data):
"""优化版本1：使用集合"""
return list(set(data))
# 优化版本2：保持顺序的去重
def optimized_v2_function(data):
"""优化版本2：保持顺序的去重"""
seen = set()
return [x for x in data if not (x in seen or seen.add(x))]
# 测试数据
test_data = list(range(1000)) * 2 + list(range(500, 1500))
# 性能测试
import time
print("测试数据长度:", len(test_data))
start_time = time.time()
result1 = unoptimized_function(test_data[:100]) # 只测试小数据量
time1 = time.time() - start_time
print(f"未优化版本（100个元素）: {time1:.4f}秒")
start_time = time.time()
result2 = optimized_v1_function(test_data)
time2 = time.time() - start_time
print(f"优化版本1（集合）: {time2:.4f}秒")
start_time = time.time()
result3 = optimized_v2_function(test_data)
time3 = time.time() - start_time
print(f"优化版本2（保持顺序）: {time3:.4f}秒")
print(f"性能提升: 集合版本比未优化版本快 {time1/time2:.1f}倍（基于100个元素的测试）")
if __name__ == "__main__":
simulate_line_profiler_analysis()
performance_optimization_tips()
optimization_example()
---
03.memory_profiler内存分析
a.memory_profiler原理
memory_profiler用于分析Python代码的内存使用情况，可以逐行显示内存消耗。帮助识别内存泄漏、内存密集型操作和优化内存使用。支持采样间隔设置和图形化显示。
b.memory_profiler内存分析示例
---
import time
import random
import gc
import sys
from collections import defaultdict
# 内存密集型操作函数
def memory_intensive_operations():
"""内存密集型操作"""
# 大列表创建
large_list = []
print("创建大列表...")
for i in range(100000):
large_list.append({
'id': i,
'data': 'x' * 100, # 100字符的字符串
'numbers': list(range(10)),
'metadata': {
'created': time.time(),
'modified': time.time()
}
})
print(f"大列表创建完成，长度: {len(large_list)}")
# 字典操作
large_dict = {}
print("创建大字典...")
for i in range(50000):
key = f"item_{i}"
value = {
'value': i * 2,
'squared': i ** 2,
'cubed': i ** 3
}
large_dict[key] = value
print(f"大字典创建完成，长度: {len(large_dict)}")
# 字符串操作
string_data = []
print("创建字符串数据...")
base_string = "This is a test string for memory profiling purposes. " * 10
for i in range(10000):
string_data.append(f"{base_string}_{i}_{random.random()}")
print(f"字符串数据创建完成，长度: {len(string_data)}")
# 模拟数据处理
print("模拟数据处理...")
processed_data = []
for i, item in enumerate(large_list[:10000]): # 只处理部分数据
processed = {
'original_id': item['id'],
'processed_value': item['id'] * 2,
'string_length': len(item['data']),
'processing_time': time.time()
}
processed_data.append(processed)
if i % 1000 == 0:
print(f"已处理 {i}/10000 项")
print(f"数据处理完成，处理了 {len(processed_data)} 项")
# 清理部分内存
print("清理部分内存...")
del large_list
del large_dict
gc.collect() # 强制垃圾回收
print("内存清理完成")
return {
'processed_count': len(processed_data),
'string_count': len(string_data),
'remaining_items': len(processed_data)
}
def memory_leak_simulation():
"""模拟内存泄漏"""
leaked_objects = []
print("模拟内存泄漏场景...")
for i in range(5):
# 创建临时对象
temp_data = [random.random() for _ in range(100000)]
processed_temp = [x * 2 for x in temp_data]
# 模拟某些对象被意外保留（泄漏）
if i % 2 == 0:
leaked_objects.extend(processed_temp[:1000])
print(f"迭代 {i+1}/5 完成，泄漏对象数量: {len(leaked_objects)}")
return len(leaked_objects)
def memory_efficient_alternatives():
"""内存高效替代方案"""
print("\n=== 内存高效替代方案演示 ===")
# 低效版本：大列表
print("低效版本：使用大列表")
start_time = time.time()
inefficient_data = []
for i in range(100000):
inefficient_data.append(i * i)
inefficient_time = time.time() - start_time
inefficient_memory = sys.getsizeof(inefficient_data)
print(f"列表创建耗时: {inefficient_time:.4f}秒，内存占用: {inefficient_memory} 字节")
# 高效版本1：生成器
print("\n高效版本1：使用生成器")
start_time = time.time()
efficient_generator = (i * i for i in range(100000))
# 消费生成器以模拟使用
generator_sum = sum(efficient_generator)
efficient_time = time.time() - start_time
print(f"生成器创建耗时: {efficient_time:.4f}秒，内存占用: 极小")
print(f"计算结果验证: {generator_sum == sum(i*i for i in range(100000))}")
# 高效版本2：数组模块
print("\n高效版本2：使用数组模块")
start_time = time.time()
from array import array
efficient_array = array('l', (i * i for i in range(100000)))
array_time = time.time() - start_time
array_memory = sys.getsizeof(efficient_array)
print(f"数组创建耗时: {array_time:.4f}秒，内存占用: {array_memory} 字节")
print(f"内存节省: {(inefficient_memory - array_memory) / inefficient_memory:.1%}")
# 高效版本3：numpy数组（如果可用）
try:
print("\n高效版本3：使用numpy数组")
import numpy as np
start_time = time.time()
numpy_array = np.arange(100000, dtype=np.int64) ** 2
numpy_time = time.time() - start_time
numpy_memory = numpy_array.nbytes
print(f"numpy数组创建耗时: {numpy_time:.4f}秒，内存占用: {numpy_memory} 字节")
print(f"相比列表节省: {(inefficient_memory - numpy_memory) / inefficient_memory:.1%}")
except ImportError:
print("numpy不可用，跳过numpy测试")
def memory_usage_patterns():
"""内存使用模式分析"""
print("\n=== 内存使用模式分析 ===")
# 模式1：累积增长
print("模式1：累积增长模式")
cumulative_data = []
memory_snapshots = []
for i in range(10):
chunk = [random.random() for _ in range(10000)]
cumulative_data.extend(chunk)
current_memory = sys.getsizeof(cumulative_data)
memory_snapshots.append(current_memory)
print(f"步骤 {i+1}: 数据项 {len(cumulative_data)}，内存 {current_memory} 字节")
print(f"内存增长率: {(memory_snapshots[-1] - memory_snapshots[0]) / memory_snapshots[0]:.1%}")
# 模式2：峰值使用
print("\n模式2：峰值使用模式")
peak_memory = 0
for i in range(5):
temp_data = [random.random() for _ in range(50000)]
current_memory = sys.getsizeof(temp_data)
peak_memory = max(peak_memory, current_memory)
print(f"步骤 {i+1}: 临时数据内存 {current_memory} 字节")
del temp_data # 释放内存
print(f"峰值内存使用: {peak_memory} 字节")
# 模式3：批量处理
print("\n模式3：批量处理模式")
batch_size = 10000
total_processed = 0
max_batch_memory = 0
for batch_start in range(0, 50000, batch_size):
batch_data = [random.random() for _ in range(batch_size)]
batch_memory = sys.getsizeof(batch_data)
max_batch_memory = max(max_batch_memory, batch_memory)
# 模拟批处理
processed_batch = [x * 2 for x in batch_data]
total_processed += len(processed_batch)
print(f"批次 {batch_start//batch_size + 1}: 处理 {len(batch_data)} 项，内存 {batch_memory} 字节")
del batch_data, processed_batch
print(f"总处理: {total_processed} 项，最大批次内存: {max_batch_memory} 字节")
def memory_optimization_techniques():
"""内存优化技术"""
print("\n=== 内存优化技术 ===")
techniques = [
{
'name': '使用生成器替代列表',
'description': '生成器按需生成数据，节省内存',
'example': 'range(1000000) vs list(range(1000000))'
},
{
'name': '及时释放大对象',
'description': '使用del语句显式释放不再需要的大对象',
'example': 'del large_list; gc.collect()'
},
{
'name': '使用__slots__减少对象内存',
'description': '在类定义中使用__slots__减少实例属性内存开销',
'example': 'class Point: __slots__ = [\'x\', \'y\']'
},
{
'name': '选择合适的数据结构',
'description': '根据使用场景选择最节省内存的数据结构',
'example': 'array.array vs list (数值数据)'
},
{
'name': '字符串intern优化',
'description': '对重复字符串使用intern技术',
'example': 'sys.intern("common_string")'
},
{
'name': '内存映射文件',
'description': '对大文件使用内存映射技术',
'example': 'mmap.mmap(file.fileno(), 0)'
}
]
for i, technique in enumerate(techniques, 1):
print(f"\n{i}. {technique['name']}")
print(f" 描述: {technique['description']}")
print(f" 示例: {technique['example']}")
def memory_monitoring_example():
"""内存监控示例"""
print("\n=== 内存监控示例 ===")
# 简单的内存监控器
def simple_memory_monitor():
"""简单内存监控"""
import os
import psutil
try:
process = psutil.Process(os.getpid())
memory_info = process.memory_info()
return {
'rss': memory_info.rss / 1024 / 1024, # MB
'vms': memory_info.vms / 1024 / 1024 # MB
}
except ImportError:
# 如果psutil不可用，使用简单方法
import resource
usage = resource.getrusage(resource.RUSAGE_SELF)
return {
'max_rss': usage.ru_maxrss / 1024 # MB (on Linux)
}
# 监控内存使用
print("开始内存监控...")
initial_memory = simple_memory_monitor()
print(f"初始内存使用: {initial_memory}")
# 执行内存密集型操作
test_data = []
for i in range(10):
chunk = [random.random() for _ in range(10000)]
test_data.extend(chunk)
current_memory = simple_memory_monitor()
print(f"步骤 {i+1}: {current_memory}")
# 简单的内存阈值检查
if isinstance(current_memory, dict) and 'rss' in current_memory:
if current_memory['rss'] > 100: # 100MB阈值
print("警告：内存使用超过100MB阈值")
break
final_memory = simple_memory_monitor()
print(f"最终内存使用: {final_memory}")
# 清理
del test_data
gc.collect()
cleaned_memory = simple_memory_monitor()
print(f"清理后内存使用: {cleaned_memory}")
# 模拟memory_profiler使用说明
def memory_profiler_usage_guide():
"""memory_profiler使用指南"""
print("\n=== Memory Profiler使用指南 ===")
print("1. 安装: pip install memory-profiler")
print("2. 在函数前添加@profile装饰器")
print("3. 运行: python -m memory_profiler script.py")
print("4. 或者: mprof run script.py")
print("5. 查看结果: mprof plot")
print("\n示例代码:")
print("""
@profile
def my_function():
a = [1] * (10 ** 6)
b = [2] * (2 * 10 ** 7)
del b
return a
if __name__ == '__main__':
my_function()
""")
if __name__ == "__main__":
print("执行内存分析演示...")
print("\n" + "="*50)
print("1. 内存密集型操作分析")
print("="*50)
memory_intensive_operations()
print("\n" + "="*50)
print("2. 内存泄漏模拟")
print("="*50)
leaked_count = memory_leak_simulation()
print(f"模拟泄漏对象总数: {leaked_count}")
print("\n" + "="*50)
print("3. 内存高效替代方案")
print("="*50)
memory_efficient_alternatives()
print("\n" + "="*50)
print("4. 内存使用模式")
print("="*50)
memory_usage_patterns()
print("\n" + "="*50)
print("5. 内存优化技术")
print("="*50)
memory_optimization_techniques()
print("\n" + "="*50)
print("6. 内存监控")
print("="*50)
memory_monitoring_example()
print("\n" + "="*50)
print("7. 使用指南")
print("="*50)
memory_profiler_usage_guide()
---
7. 异步框架
7.1 aiohttp异步HTTP
01.aiohttp基础用法
a.aiohttp原理
aiohttp是基于asyncio的异步HTTP客户端/服务器库。支持HTTP/1.1和部分HTTP/2特性。使用async/await语法处理网络请求，避免阻塞事件循环。适合高并发网络请求场景。
b.aiohttp基础示例
---
import asyncio
import aiohttp
import time
import json
from typing import Dict, List, Any
# 基础GET请求
async def basic_get_request(session, url):
"""基础GET请求"""
try:
start_time = time.time()
async with session.get(url, timeout=10) as response:
if response.status == 200:
data = await response.text()
elapsed = time.time() - start_time
return {
'url': url,
'status': response.status,
'data_length': len(data),
'elapsed': elapsed,
'headers': dict(response.headers)
}
else:
return {
'url': url,
'status': response.status,
'error': f'HTTP {response.status}',
'elapsed': time.time() - start_time
}
except Exception as e:
return {
'url': url,
'status': 0,
'error': str(e),
'elapsed': time.time() - start_time
}
# 基础POST请求
async def basic_post_request(session, url, data):
"""基础POST请求"""
try:
start_time = time.time()
headers = {'Content-Type': 'application/json'}
async with session.post(url, json=data, headers=headers, timeout=10) as response:
response_data = await response.json() if response.content_type == 'application/json' else await response.text()
elapsed = time.time() - start_time
return {
'url': url,
'method': 'POST',
'status': response.status,
'response_data': response_data,
'elapsed': elapsed
}
except Exception as e:
return {
'url': url,
'method': 'POST',
'status': 0,
'error': str(e),
'elapsed': time.time() - start_time
}
# 批量请求处理
async def batch_requests(urls):
"""批量处理HTTP请求"""
async with aiohttp.ClientSession() as session:
# 创建所有请求任务
tasks = [basic_get_request(session, url) for url in urls]
# 并发执行所有请求
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
# 限制并发数量的请求
async def limited_concurrent_requests(urls, max_concurrent=5):
"""限制并发数量的HTTP请求"""
semaphore = asyncio.Semaphore(max_concurrent)
async def limited_request(session, url):
async with semaphore:
return await basic_get_request(session, url)
async with aiohttp.ClientSession() as session:
tasks = [limited_request(session, url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
# 基础测试
async def basic_aiohttp_demo():
"""aiohttp基础演示"""
print("=== aiohttp基础演示 ===")
# 测试URL列表
urls = [
'https://httpbin.org/get',
'https://httpbin.org/delay/1',
'https://httpbin.org/status/200',
'https://jsonplaceholder.typicode.com/posts/1',
'https://api.github.com/users/octocat'
]
# 顺序请求（模拟同步）
print("\n1. 顺序请求:")
start_time = time.time()
async with aiohttp.ClientSession() as session:
sequential_results = []
for url in urls:
result = await basic_get_request(session, url)
sequential_results.append(result)
print(f" {url}: {result['status']} ({result['elapsed']:.2f}s)")
sequential_time = time.time() - start_time
print(f"顺序请求总耗时: {sequential_time:.2f}秒")
# 并发请求
print("\n2. 并发请求:")
start_time = time.time()
concurrent_results = await batch_requests(urls)
concurrent_time = time.time() - start_time
for result in concurrent_results:
if isinstance(result, dict):
print(f" {result['url']}: {result['status']} ({result['elapsed']:.2f}s)")
else:
print(f" 异常: {result}")
print(f"并发请求总耗时: {concurrent_time:.2f}秒")
print(f"性能提升: {sequential_time/concurrent_time:.2f}x")
# 限制并发请求
print("\n3. 限制并发请求（最大3个）:")
start_time = time.time()
limited_results = await limited_concurrent_requests(urls, max_concurrent=3)
limited_time = time.time() - start_time
for result in limited_results:
if isinstance(result, dict):
print(f" {result['url']}: {result['status']} ({result['elapsed']:.2f}s)")
print(f"限制并发请求总耗时: {limited_time:.2f}秒")
# POST请求测试
print("\n4. POST请求测试:")
post_data = {'name': 'test', 'value': 123}
post_url = 'https://httpbin.org/post'
async with aiohttp.ClientSession() as session:
post_result = await basic_post_request(session, post_url, post_data)
print(f"POST请求结果: {post_result['status']} ({post_result['elapsed']:.2f}s)")
if post_result['status'] == 200:
print(f" 服务器接收到的数据: {post_result['response_data'].get('json', {})}")
if __name__ == "__main__":
asyncio.run(basic_aiohttp_demo())
---
02.aiohttp高级特性
a.高级特性原理
aiohttp提供会话管理、连接池、重试机制、中间件等高级功能。支持自定义请求头、代理设置、SSL配置、超时控制。提供WebSocket支持和服务器功能。
b.aiohttp高级特性示例
---
import asyncio
import aiohttp
import time
import json
import ssl
from aiohttp import ClientSession, TCPConnector, ClientTimeout, BasicAuth
from typing import Optional, Dict, Any
class AdvancedHTTPClient:
"""高级HTTP客户端"""
def __init__(self,
max_connections: int = 100,
connection_timeout: float = 30.0,
read_timeout: float = 60.0,
total_timeout: float = 90.0):
# 配置连接器
self.connector = TCPConnector(
limit=max_connections, # 总连接池大小
limit_per_host=max_connections // 4, # 每个主机的连接数
ttl_dns_cache=300, # DNS缓存时间
use_dns_cache=True,
ssl=ssl.create_default_context(), # SSL配置
enable_cleanup_closed=True
)
# 配置超时
self.timeout = ClientTimeout(
total=total_timeout,
connect=connection_timeout,
sock_read=read_timeout
)
# 请求统计
self.stats = {
'total_requests': 0,
'successful_requests': 0,
'failed_requests': 0,
'total_time': 0.0
}
async def request_with_retry(self,
session: ClientSession,
method: str,
url: str,
max_retries: int = 3,
retry_delay: float = 1.0,
**kwargs) -> Dict[str, Any]:
"""带重试机制的HTTP请求"""
last_exception = None
for attempt in range(max_retries + 1):
try:
start_time = time.time()
async with session.request(method, url, timeout=self.timeout, **kwargs) as response:
# 根据状态码决定是否重试
if response.status in [429, 500, 502, 503, 504] and attempt < max_retries:
await asyncio.sleep(retry_delay * (2 ** attempt)) # 指数退避
continue
response_data = await self._process_response(response)
elapsed = time.time() - start_time
# 更新统计
self.stats['total_requests'] += 1
self.stats['total_time'] += elapsed
if response.status < 400:
self.stats['successful_requests'] += 1
else:
self.stats['failed_requests'] += 1
return {
'url': url,
'method': method,
'status': response.status,
'data': response_data,
'headers': dict(response.headers),
'elapsed': elapsed,
'attempt': attempt + 1
}
except Exception as e:
last_exception = e
if attempt < max_retries:
await asyncio.sleep(retry_delay * (2 ** attempt))
continue
else:
self.stats['total_requests'] += 1
self.stats['failed_requests'] += 1
break
return {
'url': url,
'method': method,
'status': 0,
'error': str(last_exception),
'elapsed': 0,
'attempt': max_retries + 1
}
async def _process_response(self, response) -> Any:
"""处理响应数据"""
content_type = response.headers.get('Content-Type', '')
if 'application/json' in content_type:
return await response.json()
elif 'text/' in content_type:
return await response.text()
else:
return await response.read()
async def batch_request(self,
requests: List[Dict[str, Any]],
max_concurrent: int = 10) -> List[Dict[str, Any]]:
"""批量请求处理"""
semaphore = asyncio.Semaphore(max_concurrent)
async def limited_request(request_info):
async with semaphore:
async with aiohttp.ClientSession(connector=self.connector) as session:
return await self.request_with_retry(
session,
request_info['method'],
request_info['url'],
**request_info.get('kwargs', {})
)
tasks = [limited_request(req) for req in requests]
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
def get_stats(self) -> Dict[str, Any]:
"""获取请求统计"""
if self.stats['total_requests'] > 0:
avg_time = self.stats['total_time'] / self.stats['total_requests']
success_rate = self.stats['successful_requests'] / self.stats['total_requests']
else:
avg_time = 0
success_rate = 0
return {
**self.stats,
'average_time': avg_time,
'success_rate': success_rate
}
# 高级客户端使用示例
async def advanced_client_demo():
"""高级客户端演示"""
print("=== aiohttp高级特性演示 ===")
client = AdvancedHTTPClient(
max_connections=50,
connection_timeout=10.0,
read_timeout=30.0
)
# 1. 基础请求测试
print("\n1. 基础请求测试:")
test_urls = [
{'method': 'GET', 'url': 'https://httpbin.org/get'},
{'method': 'GET', 'url': 'https://httpbin.org/delay/2'},
{'method': 'POST', 'url': 'https://httpbin.org/post',
'kwargs': {'json': {'test': 'data'}}}
]
for req in test_urls:
async with aiohttp.ClientSession(connector=client.connector) as session:
result = await client.request_with_retry(
session,
req['method'],
req['url'],
**req.get('kwargs', {})
)
print(f" {req['method']} {req['url']}: "
f"状态 {result['status']} (尝试 {result['attempt']} 次, "
f"耗时 {result['elapsed']:.2f}s)")
# 2. 批量请求测试
print("\n2. 批量请求测试:")
batch_requests = [
{'method': 'GET', 'url': f'https://jsonplaceholder.typicode.com/posts/{i}'}
for i in range(1, 11)
]
start_time = time.time()
batch_results = await client.batch_request(batch_requests, max_concurrent=5)
batch_time = time.time() - start_time
successful = [r for r in batch_results if isinstance(r, dict) and r['status'] == 200]
print(f"批量请求完成: {len(successful)}/{len(batch_results)} 成功, "
f"总耗时 {batch_time:.2f}s")
# 3. 请求头和认证测试
print("\n3. 请求头和认证测试:")
headers = {
'User-Agent': 'AdvancedHTTPClient/1.0',
'Accept': 'application/json',
'X-Custom-Header': 'test-value'
}
auth = BasicAuth('user', 'pass')
async with aiohttp.ClientSession(
connector=client.connector,
headers=headers,
auth=auth
) as session:
result = await client.request_with_retry(
session,
'GET',
'https://httpbin.org/get'
)
print(f" 带认证和自定义头的请求: 状态 {result['status']}")
# 4. 错误处理和重试测试
print("\n4. 错误处理和重试测试:")
error_urls = [
'https://httpbin.org/status/500', # 服务器错误
'https://httpbin.org/status/429', # 限流
'https://nonexistent-domain-12345.com' # 域名不存在
]
for url in error_urls:
async with aiohttp.ClientSession(connector=client.connector) as session:
result = await client.request_with_retry(
session,
'GET',
url,
max_retries=2,
retry_delay=0.5
)
if result['status'] == 0:
print(f" {url}: 失败 - {result['error']} (尝试 {result['attempt']} 次)")
else:
print(f" {url}: 状态 {result['status']} (尝试 {result['attempt']} 次)")
# 5. 统计信息
print("\n5. 请求统计:")
stats = client.get_stats()
print(f" 总请求数: {stats['total_requests']}")
print(f" 成功请求数: {stats['successful_requests']}")
print(f" 失败请求数: {stats['failed_requests']}")
print(f" 平均耗时: {stats['average_time']:.2f}s")
print(f" 成功率: {stats['success_rate']:.1%}")
# WebSocket客户端示例
async def websocket_client_demo():
"""WebSocket客户端演示"""
print("\n=== WebSocket客户端演示 ===")
ws_url = "wss://echo.websocket.org/"
try:
async with aiohttp.ClientSession() as session:
async with session.ws_connect(ws_url) as ws:
print(f"已连接到 {ws_url}")
# 发送消息
messages = [
"Hello WebSocket!",
"这是一条测试消息",
"WebSocket测试完成"
]
for msg in messages:
await ws.send_str(msg)
print(f"发送: {msg}")
# 接收回声
response = await ws.receive_str()
print(f"接收: {response}")
await asyncio.sleep(0.5)
print("WebSocket通信完成")
except Exception as e:
print(f"WebSocket连接失败: {e}")
# 文件下载示例
async def file_download_demo():
"""文件下载演示"""
print("\n=== 文件下载演示 ===")
download_url = "https://httpbin.org/bytes/1024" # 下载1KB的随机数据
filename = "downloaded_data.bin"
async with aiohttp.ClientSession() as session:
try:
async with session.get(download_url) as response:
if response.status == 200:
# 获取文件大小
content_length = response.headers.get('Content-Length')
print(f"开始下载，文件大小: {content_length} 字节")
# 流式下载
downloaded = 0
with open(filename, 'wb') as f:
async for chunk in response.content.iter_chunked(1024):
f.write(chunk)
downloaded += len(chunk)
if content_length:
progress = (downloaded / int(content_length)) * 100
print(f"下载进度: {progress:.1f}%")
print(f"下载完成: {filename} ({downloaded} 字节)")
# 验证下载文件
with open(filename, 'rb') as f:
downloaded_data = f.read()
print(f"文件验证通过，实际大小: {len(downloaded_data)} 字节")
else:
print(f"下载失败，状态码: {response.status}")
except Exception as e:
print(f"下载异常: {e}")
finally:
# 清理下载的文件
import os
try:
if os.path.exists(filename):
os.remove(filename)
print(f"已清理临时文件: {filename}")
except:
pass
if __name__ == "__main__":
asyncio.run(advanced_client_demo())
asyncio.run(websocket_client_demo())
asyncio.run(file_download_demo())
---
03.aiohttp性能优化
a.性能优化原理
aiohttp性能优化包括连接池复用、请求批处理、异步并发控制、缓存策略、压缩传输等。通过合理的资源管理和并发控制，最大化网络吞吐量和响应速度。
b.aiohttp性能优化示例
---
import asyncio
import aiohttp
import time
import json
import gzip
from aiohttp import ClientSession, TCPConnector, ClientTimeout
from typing import List, Dict, Any, Optional
from urllib.parse import urljoin
import hashlib
import pickle
import os
class PerformanceOptimizedClient:
"""性能优化的HTTP客户端"""
def __init__(self,
base_url: str = "",
cache_dir: str = ".http_cache",
max_cache_size: int = 1000):
self.base_url = base_url
self.cache_dir = cache_dir
self.max_cache_size = max_cache_size
# 创建缓存目录
os.makedirs(cache_dir, exist_ok=True)
# 内存缓存
self.memory_cache = {}
self.cache_stats = {
'hits': 0,
'misses': 0,
'saves': 0
}
# 优化的连接器配置
self.connector = TCPConnector(
limit=200, # 总连接池
limit_per_host=50, # 每个主机连接数
ttl_dns_cache=600, # DNS缓存10分钟
use_dns_cache=True,
keepalive_timeout=30, # 保持连接30秒
enable_cleanup_closed=True,
force_close=False, # 启用连接复用
ssl=False # 在生产环境中应该为True
)
# 超时配置
self.timeout = ClientTimeout(
total=60, # 总超时
connect=10, # 连接超时
sock_read=30 # 读取超时
)
# 请求统计
self.request_stats = {
'total_requests': 0,
'cache_hits': 0,
'total_time': 0.0,
'bytes_sent': 0,
'bytes_received': 0
}
def _get_cache_key(self, url: str, method: str = 'GET', **kwargs) -> str:
"""生成缓存键"""
cache_data = {
'url': url,
'method': method,
'params': kwargs.get('params', {}),
'headers': {k: v for k, v in kwargs.get('headers', {}).items()
if k.lower() not in ['authorization', 'cookie']}
}
cache_str = json.dumps(cache_data, sort_keys=True)
return hashlib.md5(cache_str.encode()).hexdigest()
def _get_cache_path(self, cache_key: str) -> str:
"""获取缓存文件路径"""
return os.path.join(self.cache_dir, f"{cache_key}.cache")
def _load_from_cache(self, cache_key: str) -> Optional[Dict[str, Any]]:
"""从缓存加载数据"""
# 先检查内存缓存
if cache_key in self.memory_cache:
self.cache_stats['hits'] += 1
return self.memory_cache[cache_key]
# 检查磁盘缓存
cache_path = self._get_cache_path(cache_key)
if os.path.exists(cache_path):
try:
with open(cache_path, 'rb') as f:
cached_data = pickle.load(f)
# 检查缓存是否过期（1小时）
if time.time() - cached_data['timestamp'] < 3600:
self.memory_cache[cache_key] = cached_data
self.cache_stats['hits'] += 1
return cached_data
else:
os.remove(cache_path) # 删除过期缓存
except Exception:
pass
self.cache_stats['misses'] += 1
return None
def _save_to_cache(self, cache_key: str, data: Dict[str, Any]):
"""保存数据到缓存"""
cached_data = {
'data': data,
'timestamp': time.time()
}
# 保存到内存缓存
if len(self.memory_cache) < self.max_cache_size:
self.memory_cache[cache_key] = cached_data
else:
# 清理最旧的缓存项
oldest_key = next(iter(self.memory_cache))
del self.memory_cache[oldest_key]
self.memory_cache[cache_key] = cached_data
# 保存到磁盘缓存
cache_path = self._get_cache_path(cache_key)
try:
with open(cache_path, 'wb') as f:
pickle.dump(cached_data, f)
self.cache_stats['saves'] += 1
except Exception:
pass
async def cached_request(self,
method: str,
url: str,
use_cache: bool = True,
**kwargs) -> Dict[str, Any]:
"""带缓存的HTTP请求"""
full_url = urljoin(self.base_url, url) if self.base_url else url
# 只对GET请求使用缓存
if method.upper() != 'GET' or not use_cache:
return await self._make_request(method, full_url, **kwargs)
cache_key = self._get_cache_key(full_url, method, **kwargs)
# 尝试从缓存加载
cached_data = self._load_from_cache(cache_key)
if cached_data:
self.request_stats['cache_hits'] += 1
return cached_data['data']
# 缓存未命中，发送请求
response_data = await self._make_request(method, full_url, **kwargs)
# 只缓存成功的响应
if response_data.get('status', 0) == 200:
self._save_to_cache(cache_key, response_data)
return response_data
async def _make_request(self, method: str, url: str, **kwargs) -> Dict[str, Any]:
"""执行实际的HTTP请求"""
start_time = time.time()
# 添加压缩支持
headers = kwargs.get('headers', {})
headers.setdefault('Accept-Encoding', 'gzip, deflate')
kwargs['headers'] = headers
try:
async with aiohttp.ClientSession(
connector=self.connector,
timeout=self.timeout
) as session:
async with session.request(method, url, **kwargs) as response:
# 处理响应
content_type = response.headers.get('Content-Type', '')
if 'application/json' in content_type:
data = await response.json()
elif 'text/' in content_type:
data = await response.text()
else:
data = await response.read()
elapsed = time.time() - start_time
# 更新统计
self.request_stats['total_requests'] += 1
self.request_stats['total_time'] += elapsed
self.request_stats['bytes_sent'] += len(str(kwargs.get('json', {})))
self.request_stats['bytes_received'] += len(str(data))
return {
'url': url,
'method': method,
'status': response.status,
'data': data,
'headers': dict(response.headers),
'elapsed': elapsed
}
except Exception as e:
elapsed = time.time() - start_time
return {
'url': url,
'method': method,
'status': 0,
'error': str(e),
'elapsed': elapsed
}
async def batch_requests_optimized(self,
requests: List[Dict[str, Any]],
max_concurrent: int = 50,
batch_size: int = 100) -> List[Dict[str, Any]]:
"""优化的批量请求处理"""
all_results = []
# 分批处理大量请求
for i in range(0, len(requests), batch_size):
batch = requests[i:i + batch_size]
print(f"处理批次 {i//batch_size + 1}/{(len(requests)-1)//batch_size + 1} "
f"({len(batch)} 个请求)")
# 信号量控制并发
semaphore = asyncio.Semaphore(max_concurrent)
async def limited_request(req_info):
async with semaphore:
return await self.cached_request(
req_info['method'],
req_info['url'],
**req_info.get('kwargs', {})
)
# 执行当前批次
start_time = time.time()
tasks = [limited_request(req) for req in batch]
batch_results = await asyncio.gather(*tasks, return_exceptions=True)
batch_time = time.time() - start_time
all_results.extend(batch_results)
successful = sum(1 for r in batch_results
if isinstance(r, dict) and r.get('status', 0) == 200)
print(f"批次完成: {successful}/{len(batch)} 成功, 耗时 {batch_time:.2f}s")
# 批次间短暂休息，避免过度请求
await asyncio.sleep(0.1)
return all_results
def get_performance_stats(self) -> Dict[str, Any]:
"""获取性能统计信息"""
stats = self.request_stats.copy()
if stats['total_requests'] > 0:
stats['average_time'] = stats['total_time'] / stats['total_requests']
stats['requests_per_second'] = stats['total_requests'] / max(stats['total_time'], 1)
stats['cache_hit_rate'] = stats['cache_hits'] / stats['total_requests']
else:
stats['average_time'] = 0
stats['requests_per_second'] = 0
stats['cache_hit_rate'] = 0
stats.update(self.cache_stats)
return stats
def clear_cache(self):
"""清理缓存"""
# 清理内存缓存
self.memory_cache.clear()
# 清理磁盘缓存
try:
for filename in os.listdir(self.cache_dir):
if filename.endswith('.cache'):
os.remove(os.path.join(self.cache_dir, filename))
except Exception:
pass
# 重置统计
self.cache_stats = {'hits': 0, 'misses': 0, 'saves': 0}
# 性能测试
async def performance_test():
"""性能测试"""
print("=== aiohttp性能优化测试 ===")
# 创建优化客户端
client = PerformanceOptimizedClient(
base_url="https://jsonplaceholder.typicode.com"
)
# 准备测试请求
test_requests = [
{'method': 'GET', 'url': f'/posts/{i}'}
for i in range(1, 101) # 100个请求
]
print(f"准备执行 {len(test_requests)} 个请求...")
# 第一次执行（无缓存）
print("\n1. 第一次执行（缓存预热）:")
start_time = time.time()
results1 = await client.batch_requests_optimized(
test_requests,
max_concurrent=20,
batch_size=50
)
first_time = time.time() - start_time
successful1 = sum(1 for r in results1 if isinstance(r, dict) and r.get('status') == 200)
print(f"第一次执行完成: {successful1}/{len(test_requests)} 成功, 耗时 {first_time:.2f}s")
# 第二次执行（使用缓存）
print("\n2. 第二次执行（使用缓存）:")
start_time = time.time()
results2 = await client.batch_requests_optimized(
test_requests,
max_concurrent=20,
batch_size=50
)
second_time = time.time() - start_time
successful2 = sum(1 for r in results2 if isinstance(r, dict) and r.get('status') == 200)
print(f"第二次执行完成: {successful2}/{len(test_requests)} 成功, 耗时 {second_time:.2f}s")
print(f"缓存加速比: {first_time/second_time:.1f}x")
# 性能统计
print("\n3. 性能统计:")
stats = client.get_performance_stats()
print(f" 总请求数: {stats['total_requests']}")
print(f" 平均响应时间: {stats['average_time']:.3f}s")
print(f" 请求速率: {stats['requests_per_second']:.1f} req/s")
print(f" 缓存命中率: {stats['cache_hit_rate']:.1%}")
print(f" 数据传输: 发送 {stats['bytes_sent']} 字节, 接收 {stats['bytes_received']} 字节")
# 连接池复用测试
print("\n4. 连接池复用测试:")
reuse_test_requests = [
{'method': 'GET', 'url': 'https://httpbin.org/get'} for _ in range(20)
]
async with aiohttp.ClientSession(connector=client.connector) as session:
start_time = time.time()
reuse_tasks = [
client._make_request('GET', 'https://httpbin.org/get')
for _ in range(20)
]
reuse_results = await asyncio.gather(*reuse_tasks)
reuse_time = time.time() - start_time
reuse_successful = sum(1 for r in reuse_results if isinstance(r, dict) and r.get('status') == 200)
print(f"连接池复用测试: {reuse_successful}/20 成功, 耗时 {reuse_time:.2f}s")
print(f"平均单请求时间: {reuse_time/20:.3f}s")
# 清理缓存
print("\n5. 清理缓存:")
client.clear_cache()
print("缓存清理完成")
# 并发压力测试
async def stress_test():
"""并发压力测试"""
print("\n=== 并发压力测试 ===")
client = PerformanceOptimizedClient()
# 不同并发级别的测试
concurrency_levels = [1, 5, 10, 20, 50]
test_url = 'https://httpbin.org/delay/0.1'
requests_per_level = 50
results = {}
for concurrency in concurrency_levels:
print(f"\n测试并发级别: {concurrency}")
# 创建请求列表
requests = [
{'method': 'GET', 'url': test_url}
for _ in range(requests_per_level)
]
# 执行测试
start_time = time.time()
test_results = await client.batch_requests_optimized(
requests,
max_concurrent=concurrency
)
test_time = time.time() - start_time
# 统计结果
successful = sum(1 for r in test_results if isinstance(r, dict) and r.get('status') == 200)
avg_time = sum(r.get('elapsed', 0) for r in test_results if isinstance(r, dict)) / len(test_results)
results[concurrency] = {
'successful': successful,
'total_time': test_time,
'avg_time': avg_time,
'throughput': successful / test_time
}
print(f" 成功: {successful}/{requests_per_level}")
print(f" 总耗时: {test_time:.2f}s")
print(f" 平均响应时间: {avg_time:.3f}s")
print(f" 吞吐量: {successful/test_time:.1f} req/s")
# 找出最优并发级别
best_concurrency = max(results.keys(), key=lambda k: results[k]['throughput'])
print(f"\n最优并发级别: {best_concurrency} (吞吐量: {results[best_concurrency]['throughput']:.1f} req/s)")
if __name__ == "__main__":
asyncio.run(performance_test())
asyncio.run(stress_test())
---
7.2 aiofiles异步文件
01.异步文件操作概述
a.基本介绍
aiofiles是一个专门为Python异步编程设计的文件操作库，提供了基于asyncio的异步文件读写功能。
它利用asyncio事件循环的优势，在文件IO操作时不会阻塞事件循环，提高了应用程序的并发性能。
aiofiles的设计目标是提供与标准文件操作相似的API，但完全异步化，支持文件的打开、读取、写入、关闭等操作。
与传统同步文件操作相比，aiofiles特别适合于需要同时处理多个文件IO操作的场景，如Web服务器、数据分析管道等。
b.核心优势
a.非阻塞IO
异步文件操作不会阻塞事件循环，允许其他协程在文件IO等待期间继续执行。
b.简单易用的API
提供与Python内置文件操作相似的接口，学习成本低，迁移容易。
c.与asyncio生态兼容
完全兼容asyncio框架，可以与异步HTTP服务器、数据库操作等无缝集成。
d.跨平台支持
支持Windows、Linux、macOS等多个操作系统，提供一致的异步文件操作体验。
c.代码示例
---
# aiofiles基础异步文件操作示例
import asyncio
import aiofiles
import os
from pathlib import Path
# GLM智能：异步文件写入演示
async def write_file_demo():
"""演示异步文件写入操作"""
# 异步创建目录
os.makedirs("demo_files", exist_ok=True)
# 异步写入文件
async with aiofiles.open("demo_files/async_demo.txt", mode='w', encoding='utf-8') as file:
await file.write("这是一个异步文件写入演示\n")
await file.write("使用aiofiles库进行操作\n")
await file.write("写入操作不会阻塞事件循环\n")
print("文件写入完成")
# GLM智能：异步文件读取演示
async def read_file_demo():
"""演示异步文件读取操作"""
try:
# 异步读取文件
async with aiofiles.open("demo_files/async_demo.txt", mode='r', encoding='utf-8') as file:
content = await file.read()
print("文件内容读取成功:")
print(content)
return content
except FileNotFoundError:
print("文件不存在")
return None
# GLM智能：异步文件逐行读取演示
async def read_lines_demo():
"""演示异步逐行读取文件"""
try:
lines = []
async with aiofiles.open("demo_files/async_demo.txt", mode='r', encoding='utf-8') as file:
async for line in file:
lines.append(line.strip())
print("逐行读取完成:")
for i, line in enumerate(lines, 1):
print(f"第{i}行: {line}")
return lines
except Exception as e:
print(f"读取文件时出错: {e}")
return []
# GLM智能：异步文件复制演示
async def copy_file_demo():
"""演示异步文件复制操作"""
try:
# 异步读取源文件
async with aiofiles.open("demo_files/async_demo.txt", mode='rb') as source:
content = await source.read()
# 异步写入目标文件
async with aiofiles.open("demo_files/copy_demo.txt", mode='wb') as target:
await target.write(content)
print("文件复制完成")
except Exception as e:
print(f"文件复制失败: {e}")
# GLM智能：异步文件信息获取演示
async def file_info_demo():
"""演示异步获取文件信息"""
from pathlib import Path
file_path = Path("demo_files/async_demo.txt")
try:
# 使用aiofiles获取文件大小
async with aiofiles.open(file_path, mode='rb') as file:
# 移动到文件末尾获取大小
await file.seek(0, 2) # SEEK_END
size = await file.tell()
await file.seek(0) # 重置到文件开头
# 获取其他文件信息
stat_info = file_path.stat()
print(f"文件信息:")
print(f" 文件路径: {file_path}")
print(f" 文件大小: {size} 字节")
print(f" 修改时间: {stat_info.st_mtime}")
print(f" 文件权限: {oct(stat_info.st_mode)[-3:]}")
return {
'path': str(file_path),
'size': size,
'modified': stat_info.st_mtime,
'permissions': oct(stat_info.st_mode)[-3:]
}
except Exception as e:
print(f"获取文件信息失败: {e}")
return None
# GLM智能：批量文件处理演示
async def batch_files_demo():
"""演示批量异步文件处理"""
import json
# 创建多个测试文件
test_files = [
"demo_files/data1.json",
"demo_files/data2.json",
"demo_files/data3.json"
]
# 准备测试数据
test_data = [
{"id": 1, "name": "Alice", "age": 25},
{"id": 2, "name": "Bob", "age": 30},
{"id": 3, "name": "Charlie", "age": 35}
]
# GLM智能：批量写入文件
write_tasks = []
for i, (file_path, data) in enumerate(zip(test_files, test_data)):
task = write_json_file(file_path, data)
write_tasks.append(task)
# 并发执行所有写入操作
await asyncio.gather(*write_tasks)
print("批量文件写入完成")
# GLM智能：批量读取文件
read_tasks = []
for file_path in test_files:
task = read_json_file(file_path)
read_tasks.append(task)
# 并发执行所有读取操作
results = await asyncio.gather(*read_tasks, return_exceptions=True)
print("批量文件读取完成:")
for i, (file_path, result) in enumerate(zip(test_files, results)):
if isinstance(result, Exception):
print(f" {file_path}: 读取失败 - {result}")
else:
print(f" {file_path}: {result}")
# GLM智能：辅助函数 - 异步JSON文件写入
async def write_json_file(file_path: str, data: dict):
"""异步写入JSON文件"""
try:
json_str = json.dumps(data, ensure_ascii=False, indent=2)
async with aiofiles.open(file_path, mode='w', encoding='utf-8') as file:
await file.write(json_str)
return True
except Exception as e:
print(f"写入JSON文件失败 {file_path}: {e}")
return False
# GLM智能：辅助函数 - 异步JSON文件读取
async def read_json_file(file_path: str):
"""异步读取JSON文件"""
try:
async with aiofiles.open(file_path, mode='r', encoding='utf-8') as file:
content = await file.read()
return json.loads(content)
except Exception as e:
print(f"读取JSON文件失败 {file_path}: {e}")
return None
# GLM智能：主函数演示
async def main():
"""主演示函数"""
print("=== aiofiles异步文件操作演示 ===\n")
# 1. 文件写入演示
print("1. 异步文件写入:")
await write_file_demo()
print()
# 2. 文件读取演示
print("2. 异步文件读取:")
await read_file_demo()
print()
# 3. 逐行读取演示
print("3. 异步逐行读取:")
await read_lines_demo()
print()
# 4. 文件复制演示
print("4. 异步文件复制:")
await copy_file_demo()
print()
# 5. 文件信息演示
print("5. 异步文件信息:")
await file_info_demo()
print()
# 6. 批量处理演示
print("6. 异步批量处理:")
await batch_files_demo()
print()
print("=== aiofiles演示完成 ===")
# 运行演示
if __name__ == "__main__":
asyncio.run(main())
---
7.3 asyncpg异步数据库
01.异步数据库概述
a.基本介绍
asyncpg是专门为Python异步编程设计的PostgreSQL数据库驱动，提供了完整的异步数据库操作接口。
它基于libpq库实现，支持PostgreSQL的所有主要功能，包括连接管理、事务处理、预编译语句等。
asyncpg的优势在于它完全异步化，不会阻塞事件循环，适合于高并发的Web应用和数据分析场景。
与传统的同步数据库驱动相比，asyncpg能够在数据库操作等待期间释放事件循环，让其他协程继续执行，显著提升应用性能。
b.核心特性
a.完全异步支持
所有数据库操作都是异步的，包括连接、查询、事务、预编译语句等。
b.高性能
通过连接池和预编译语句优化，提供优秀的性能表现。
c.功能完整
支持PostgreSQL的所有高级特性，如JSON支持、数组类型、COPY命令等。
d.类型安全
自动将Python类型映射到PostgreSQL类型，确保数据类型安全。
c.代码示例
---
# asyncpg异步PostgreSQL操作示例
import asyncio
import asyncpg
import json
from datetime import datetime
from typing import List, Dict, Any, Optional
# GLM智能：数据库连接池管理
class DatabaseManager:
def __init__(self, connection_string: str):
self.connection_string = connection_string
self.pool = None
async def initialize_pool(self):
"""初始化连接池"""
try:
self.pool = await asyncpg.create_pool(
self.connection_string,
min_size=5,
max_size=20,
command_timeout=60
)
print("数据库连接池初始化成功")
except Exception as e:
print(f"连接池初始化失败: {e}")
raise
async def close_pool(self):
"""关闭连接池"""
if self.pool:
await self.pool.close()
print("数据库连接池已关闭")
async def get_connection(self):
"""获取数据库连接"""
if not self.pool:
await self.initialize_pool()
return self.pool.acquire()
async def release_connection(self, connection):
"""释放数据库连接"""
if self.pool:
await self.pool.release(connection)
# GLM智能：用户模型操作
class UserOperations:
def __init__(self, db_manager: DatabaseManager):
self.db = db_manager
async def create_user_table(self):
"""创建用户表"""
conn = await self.db.get_connection()
try:
await conn.execute('''
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
username VARCHAR(50) UNIQUE NOT NULL,
email VARCHAR(100) UNIQUE NOT NULL,
age INTEGER,
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
print("用户表创建成功")
except Exception as e:
print(f"创建用户表失败: {e}")
finally:
await self.db.release_connection(conn)
async def insert_user(self, username: str, email: str, age: int = None) -> int:
"""插入用户记录"""
conn = await self.db.get_connection()
try:
# 使用参数化查询防止SQL注入
result = await conn.fetchval('''
INSERT INTO users (username, email, age)
VALUES ($1, $2, $3)
RETURNING id
''', username, email, age)
print(f"用户 {username} 插入成功，ID: {result}")
return result
except Exception as e:
print(f"插入用户失败: {e}")
return 0
finally:
await self.db.release_connection(conn)
async def get_user_by_id(self, user_id: int) -> Optional[Dict[str, Any]]:
"""根据ID获取用户信息"""
conn = await self.db.get_connection()
try:
row = await conn.fetchrow('''
SELECT id, username, email, age, is_active, created_at, updated_at
FROM users
WHERE id = $1
''', user_id)
if row:
user_info = {
'id': row['id'],
'username': row['username'],
'email': row['email'],
'age': row['age'],
'is_active': row['is_active'],
'created_at': row['created_at'],
'updated_at': row['updated_at']
}
print(f"找到用户: {user_info}")
return user_info
else:
print(f"用户ID {user_id} 不存在")
return None
except Exception as e:
print(f"查询用户失败: {e}")
return None
finally:
await self.db.release_connection(conn)
async def update_user(self, user_id: int, **kwargs) -> bool:
"""更新用户信息"""
if not kwargs:
return False
# 构建动态更新语句
set_clauses = []
values = []
for i, (key, value) in enumerate(kwargs.items(), 1):
set_clauses.append(f"{key} = ${i}")
values.append(value)
set_clauses.append("updated_at = CURRENT_TIMESTAMP")
values.append(user_id)
conn = await self.db.get_connection()
try:
sql = f'''
UPDATE users
SET {', '.join(set_clauses)}
WHERE id = ${len(values)}
'''
await conn.execute(sql, *values)
print(f"用户 {user_id} 更新成功")
return True
except Exception as e:
print(f"更新用户失败: {e}")
return False
finally:
await self.db.release_connection(conn)
async def delete_user(self, user_id: int) -> bool:
"""删除用户记录"""
conn = await self.db.get_connection()
try:
result = await conn.execute('DELETE FROM users WHERE id = $1', user_id)
if result == 'DELETE 1':
print(f"用户 {user_id} 删除成功")
return True
else:
print(f"用户 {user_id} 不存在")
return False
except Exception as e:
print(f"删除用户失败: {e}")
return False
finally:
await self.db.release_connection(conn)
async def get_all_users(self) -> List[Dict[str, Any]]:
"""获取所有用户"""
conn = await self.db.get_connection()
try:
rows = await conn.fetch('''
SELECT id, username, email, age, is_active, created_at, updated_at
FROM users
ORDER BY created_at DESC
''')
users = []
for row in rows:
user_info = {
'id': row['id'],
'username': row['username'],
'email': row['email'],
'age': row['age'],
'is_active': row['is_active'],
'created_at': row['created_at'],
'updated_at': row['updated_at']
}
users.append(user_info)
print(f"找到 {len(users)} 个用户")
return users
except Exception as e:
print(f"查询用户列表失败: {e}")
return []
finally:
await self.db.release_connection(conn)
# GLM智能：批量操作演示
class BatchOperations:
def __init__(self, db_manager: DatabaseManager):
self.db = db_manager
async def batch_insert_users(self, users_data: List[Dict[str, Any]]) -> List[int]:
"""批量插入用户"""
conn = await self.db.get_connection()
try:
# 使用executemany进行批量插入
await conn.executemany('''
INSERT INTO users (username, email, age)
VALUES ($1, $2, $3)
''', [(user['username'], user['email'], user.get('age')) for user in users_data])
print(f"批量插入 {len(users_data)} 个用户成功")
return [user.get('id') for user in users_data]
except Exception as e:
print(f"批量插入失败: {e}")
return []
finally:
await self.db.release_connection(conn)
async def batch_update_ages(self, age_updates: Dict[int, int]) -> int:
"""批量更新年龄"""
conn = await self.db.get_connection()
try:
# 逐个更新，避免复杂的事务逻辑
for user_id, new_age in age_updates.items():
await conn.execute('''
UPDATE users
SET age = $1, updated_at = CURRENT_TIMESTAMP
WHERE id = $2
''', new_age, user_id)
print(f"批量更新 {len(age_updates)} 个用户年龄")
return len(age_updates)
except Exception as e:
print(f"批量更新失败: {e}")
return 0
finally:
await self.db.release_connection(conn)
# GLM智能：事务处理演示
class TransactionDemo:
def __init__(self, db_manager: DatabaseManager):
self.db = db_manager
async def transfer_user_data(self, from_user_id: int, to_user_id: int,
transfer_type: str, amount: float) -> bool:
"""用户数据转移事务演示"""
conn = await self.db.get_connection()
try:
# 开始事务
async with conn.transaction():
# 检查源用户是否存在且有足够权限
source_user = await conn.fetchrow('''
SELECT id, username, is_active FROM users
WHERE id = $1 FOR UPDATE
''', from_user_id)
if not source_user:
print(f"源用户 {from_user_id} 不存在")
return False
if not source_user['is_active']:
print(f"源用户 {source_user['username']} 未激活")
return False
# 检查目标用户是否存在且激活
target_user = await conn.fetchrow('''
SELECT id, username, is_active FROM users
WHERE id = $1 FOR UPDATE
''', to_user_id)
if not target_user:
print(f"目标用户 {to_user_id} 不存在")
return False
if not target_user['is_active']:
print(f"目标用户 {target_user['username']} 未激活")
return False
# 在这里添加具体的转移逻辑
# 例如：权限转移、数据复制等
print(f"正在从用户 {source_user['username']} 转移 {transfer_type} 到用户 {target_user['username']}")
print(f"转移金额/数量: {amount}")
# 记录转移日志
await conn.execute('''
INSERT INTO transfer_logs
(from_user_id, to_user_id, transfer_type, amount, created_at)
VALUES ($1, $2, $3, $4, CURRENT_TIMESTAMP)
''', from_user_id, to_user_id, transfer_type, amount)
print("数据转移完成")
return True
except Exception as e:
print(f"数据转移失败: {e}")
return False
finally:
await self.db.release_connection(conn)
# GLM智能：主函数演示
async def main():
"""主演示函数"""
# 数据库连接字符串（请根据实际情况修改）
CONNECTION_STRING = "postgresql://username:password@localhost:5432/dbname"
print("=== asyncpg异步PostgreSQL操作演示 ===\n")
# 1. 初始化数据库管理器
db_manager = DatabaseManager(CONNECTION_STRING)
try:
# 2. 初始化连接池
print("1. 初始化数据库连接池:")
await db_manager.initialize_pool()
print()
# 3. 创建操作实例
user_ops = UserOperations(db_manager)
batch_ops = BatchOperations(db_manager)
trans_demo = TransactionDemo(db_manager)
# 4. 创建表结构
print("2. 创建用户表:")
await user_ops.create_user_table()
print()
# 5. 插入测试数据
print("3. 插入测试用户:")
users_to_insert = [
{"username": "alice", "email": "alice@example.com", "age": 25},
{"username": "bob", "email": "bob@example.com", "age": 30},
{"username": "charlie", "email": "charlie@example.com", "age": 35},
]
inserted_ids = await batch_ops.batch_insert_users(users_to_insert)
print(f"插入的用户ID: {inserted_ids}")
print()
# 6. 查询用户信息
print("4. 查询用户信息:")
all_users = await user_ops.get_all_users()
print()
# 7. 更新用户信息
print("5. 更新用户信息:")
if inserted_ids:
await user_ops.update_user(
inserted_ids[0],
age=26,
is_active=False
)
print()
# 8. 事务演示
print("6. 事务操作演示:")
if len(inserted_ids) >= 2:
await trans_demo.transfer_user_data(
inserted_ids[0],
inserted_ids[1],
"permission",
100.0
)
print()
# 9. 查询更新后的数据
print("7. 查询更新后的数据:")
updated_users = await user_ops.get_all_users()
print()
except Exception as e:
print(f"演示过程中出现错误: {e}")
finally:
# 10. 关闭连接池
print("8. 关闭数据库连接池:")
await db_manager.close_pool()
print("=== asyncpg演示完成 ===")
# 运行演示（需要真实的PostgreSQL数据库）
# if __name__ == "__main__":
# asyncio.run(main())
---
7.4 异步生成器
01.异步生成器概述
a.基本介绍
异步生成器是Python 3.6引入的特性，它结合了生成器和异步编程的优点，允许在生成器函数中使用await表达式。
异步生成器使用async def语法定义，使用yield而不是return来返回值，每次yield都会暂停执行并等待下一次调用。
与普通生成器不同，异步生成器的__anext__方法是一个协程，需要使用await来获取下一个值，这使得它可以在生成过程中执行异步操作。
异步生成器特别适合处理异步数据流、无限数据序列、或者需要逐步生成结果的复杂异步操作，如数据库查询、网络请求等场景。
b.核心特性
a.异步暂停与恢复
生成器函数可以在yield点暂停执行，允许事件循环处理其他任务，然后在需要时恢复执行。
b.异步IO操作支持
可以在生成器函数内部使用await表达式，执行异步IO操作而不阻塞整个程序。
c.内存效率优化
按需生成数据而不是一次性生成所有数据，节省内存空间，特别适合大数据集处理。
d.惰性计算支持
支持惰性计算模式，只在需要时才计算和生成值，提高程序性能。
c.代码示例
---
# GLM智能：异步生成器基础示例
import asyncio
import random
import aiofiles
from datetime import datetime
from typing import AsyncGenerator, List, Dict, Any
# GLM智能：基本异步生成器演示
async def basic_async_generator() -> AsyncGenerator[int, None]:
"""基本的异步生成器示例"""
for i in range(5):
# 模拟异步操作
await asyncio.sleep(0.1)
print(f"生成值 {i} 在 {datetime.now().strftime('%H:%M:%S')}")
yield i * 2
# GLM智能：异步数据流生成器
async def data_stream_generator(total_items: int, delay: float = 0.1) -> AsyncGenerator[Dict[str, Any], None]:
"""模拟异步数据流生成器"""
for i in range(total_items):
# 模拟异步数据获取
await asyncio.sleep(delay)
# 生成模拟数据
data = {
'id': i + 1,
'timestamp': datetime.now().isoformat(),
'value': random.randint(1, 100),
'status': random.choice(['active', 'pending', 'completed'])
}
print(f"生成数据项 {data['id']}: {data}")
yield data
# GLM智能：异步文件读取生成器
async def file_line_generator(file_path: str) -> AsyncGenerator[str, None]:
"""逐行异步读取文件的生成器"""
async with aiofiles.open(file_path, mode='r', encoding='utf-8') as file:
async for line in file:
# 模拟处理延迟
await asyncio.sleep(0.05)
yield line.strip()
# GLM智能：异步数据库查询生成器
async def database_query_generator(query: str, batch_size: int = 10) -> AsyncGenerator[List[Dict], None]:
"""模拟异步数据库查询生成器"""
offset = 0
while True:
# 模拟异步数据库查询
await asyncio.sleep(0.2) # 模拟网络延迟
# 模拟查询结果
if offset >= 50: # 假设总共50条记录
break
batch_data = []
for i in range(batch_size):
if offset + i >= 50:
break
record = {
'id': offset + i + 1,
'name': f'User_{offset + i + 1}',
'age': random.randint(18, 65),
'email': f'user{offset + i + 1}@example.com'
}
batch_data.append(record)
print(f"生成批次 {offset // batch_size + 1}: {len(batch_data)} 条记录")
yield batch_data
offset += batch_size
# GLM智能：异步无限序列生成器
async def infinite_sequence_generator(start: int = 0) -> AsyncGenerator[int, None]:
"""生成无限序列的异步生成器"""
current = start
while True:
await asyncio.sleep(0.1)
yield current
current += 1
# GLM智能：异步斐波那契数列生成器
async def fibonacci_generator(limit: int = 100) -> AsyncGenerator[int, None]:
"""异步生成斐波那契数列"""
a, b = 0, 1
while a <= limit:
await asyncio.sleep(0.05)
yield a
a, b = b, a + b
# GLM智能：异步生成器消费者演示
async def basic_generator_consumer():
"""基本异步生成器消费者"""
print("=== 基本异步生成器演示 ===")
async for value in basic_async_generator():
print(f"消费值: {value}")
print("基本生成器演示完成\n")
# GLM智能：数据流处理演示
async def data_stream_consumer():
"""数据流处理消费者"""
print("=== 异步数据流处理演示 ===")
processed_count = 0
async for data in data_stream_generator(10, 0.2):
# 模拟数据处理
await asyncio.sleep(0.1)
# 处理数据
processed_data = {
**data,
'processed': True,
'processing_time': datetime.now().isoformat()
}
print(f"处理完成: {processed_data}")
processed_count += 1
print(f"数据流处理完成，共处理 {processed_count} 条数据\n")
# GLM智能：批量数据处理演示
async def batch_data_consumer():
"""批量数据处理消费者"""
print("=== 异步批量数据处理演示 ===")
total_records = 0
batch_count = 0
async for batch in database_query_generator("SELECT * FROM users", 8):
batch_count += 1
records_in_batch = len(batch)
total_records += records_in_batch
print(f"处理批次 {batch_count}: {records_in_batch} 条记录")
# 模拟批次处理
await asyncio.sleep(0.1)
# 显示处理进度
print(f"累计处理: {total_records} 条记录")
print(f"批量处理完成，总计 {total_records} 条记录，分 {batch_count} 个批次\n")
# GLM智能：有限序列演示
async def fibonacci_consumer():
"""斐波那契数列消费者"""
print("=== 异步斐波那契数列演示 ===")
fib_sequence = []
async for number in fibonacci_generator(50):
fib_sequence.append(number)
print(f"斐波那契数: {number}")
print(f"完整序列: {fib_sequence}")
print(f"数列长度: {len(fib_sequence)}\n")
# GLM智能：异步生成器管道演示
async def generator_pipeline_demo():
"""异步生成器管道演示"""
print("=== 异步生成器管道演示 ===")
async def source_generator() -> AsyncGenerator[int, None]:
"""数据源生成器"""
for i in range(10):
await asyncio.sleep(0.05)
yield i
async def filter_generator(source: AsyncGenerator[int, None]) -> AsyncGenerator[int, None]:
"""过滤生成器 - 只要偶数"""
async for value in source:
await asyncio.sleep(0.02)
if value % 2 == 0:
yield value
async def transform_generator(source: AsyncGenerator[int, None]) -> AsyncGenerator[str, None]:
"""转换生成器 - 将数字转换为字符串"""
async for value in source:
await asyncio.sleep(0.02)
yield f"处理后的值: {value * 10}"
# 创建管道
source = source_generator()
filtered = filter_generator(source)
transformed = transform_generator(filtered)
# 消费管道输出
results = []
async for result in transformed:
print(f"管道输出: {result}")
results.append(result)
print(f"管道处理完成，共生成 {len(results)} 个结果\n")
# GLM智能：多生成器并发演示
async def concurrent_generators_demo():
"""多个异步生成器并发演示"""
print("=== 并发异步生成器演示 ===")
async def generator_a(name: str) -> AsyncGenerator[str, None]:
"""生成器A"""
for i in range(3):
await asyncio.sleep(0.2)
yield f"{name}-A-{i}"
async def generator_b(name: str) -> AsyncGenerator[str, None]:
"""生成器B"""
for i in range(3):
await asyncio.sleep(0.15)
yield f"{name}-B-{i}"
# 创建多个生成器
gen_a = generator_a("Test")
gen_b = generator_b("Test")
# 并发消费多个生成器
tasks = []
async for value in gen_a:
tasks.append(("A", value))
async for value in gen_b:
tasks.append(("B", value))
# 显示结果
for gen_type, value in tasks:
print(f"生成器 {gen_type}: {value}")
print(f"并发处理完成，共 {len(tasks)} 个值\n")
# GLM智能：异步生成器异常处理演示
async def error_handling_demo():
"""异步生成器异常处理演示"""
print("=== 异步生成器异常处理演示 ===")
async def faulty_generator() -> AsyncGenerator[int, None]:
"""可能出错的生成器"""
for i in range(5):
await asyncio.sleep(0.1)
if i == 3:
raise ValueError("模拟错误：处理到第3个值时出现问题")
yield i
try:
async for value in faulty_generator():
print(f"正常值: {value}")
except ValueError as e:
print(f"捕获异常: {e}")
except Exception as e:
print(f"未知异常: {e}")
print("异常处理演示完成\n")
# GLM智能：异步生成器资源管理演示
async def resource_management_demo():
"""异步生成器资源管理演示"""
print("=== 异步生成器资源管理演示 ===")
async def resource_intensive_generator() -> AsyncGenerator[str, None]:
"""资源密集型生成器"""
resource_acquired = False
try:
# 模拟资源获取
print("获取资源...")
await asyncio.sleep(0.1)
resource_acquired = True
print("资源获取成功")
# 生成数据
for i in range(5):
await asyncio.sleep(0.1)
yield f"数据-{i}"
except Exception as e:
print(f"生成器异常: {e}")
finally:
# 确保资源释放
if resource_acquired:
print("释放资源...")
await asyncio.sleep(0.05)
print("资源释放完成")
# 使用资源管理生成器
async for data in resource_intensive_generator():
print(f"消费数据: {data}")
print("资源管理演示完成\n")
# GLM智能：主演示函数
async def main():
"""主演示函数"""
print("=== 异步生成器完整演示 ===\n")
# 1. 基本生成器演示
await basic_generator_consumer()
# 2. 数据流处理演示
await data_stream_consumer()
# 3. 批量数据处理演示
await batch_data_consumer()
# 4. 斐波那契数列演示
await fibonacci_consumer()
# 5. 生成器管道演示
await generator_pipeline_demo()
# 6. 并发生成器演示
await concurrent_generators_demo()
# 7. 异常处理演示
await error_handling_demo()
# 8. 资源管理演示
await resource_management_demo()
print("=== 异步生成器演示完成 ===")
# 运行演示
if __name__ == "__main__":
asyncio.run(main())
---
7.5 异步迭代器
01.异步迭代器概述
a.基本介绍
异步迭代器是Python异步编程中的重要概念，它允许在迭代过程中执行异步操作，为处理异步数据序列提供了优雅的解决方案。
异步迭代器实现了__aiter__和__anext__两个魔术方法，其中__aiter__返回迭代器对象本身，__anext__返回异步的下一个值。
通过async for语法，可以方便地遍历异步迭代器，每次迭代都会等待__anext__方法完成，这使其特别适合处理需要异步IO操作的数据源。
异步迭代器广泛应用于文件读取、网络请求、数据库查询、消息队列等场景，能够在不阻塞事件循环的情况下处理数据流。
b.核心特性
a.异步迭代协议
实现了异步迭代协议(__aiter__和__anext__)，支持async for语法进行异步遍历。
b.非阻塞操作
在每次迭代过程中可以执行异步操作，不会阻塞事件循环，保持程序的响应性。
c.资源自动管理
支持async with语句，可以自动管理资源的获取和释放，确保资源的正确清理。
d.异常处理机制
完善的异常处理机制，支持在迭代过程中正确处理和传播异常。
c.代码示例
---
# GLM智能：异步迭代器完整示例
import asyncio
import aiohttp
import aiofiles
import json
import random
from datetime import datetime
from typing import AsyncIterator, List, Dict, Any, Optional
# GLM智能：基础异步迭代器类
class AsyncCounter:
"""基础异步计数器迭代器"""
def __init__(self, start: int = 0, end: int = 10, delay: float = 0.1):
self.current = start
self.end = end
self.delay = delay
def __aiter__(self) -> AsyncIterator[int]:
"""返回异步迭代器对象"""
return self
async def __anext__(self) -> int:
"""异步获取下一个值"""
if self.current >= self.end:
raise StopAsyncIteration
# 模拟异步操作
await asyncio.sleep(self.delay)
value = self.current
self.current += 1
return value
# GLM智能：异步文件迭代器
class AsyncFileReader:
"""异步文件读取迭代器"""
def __init__(self, file_path: str, chunk_size: int = 1024):
self.file_path = file_path
self.chunk_size = chunk_size
self.file_handle = None
async def __aenter__(self):
"""异步上下文管理器入口"""
self.file_handle = await aiofiles.open(self.file_path, mode='r', encoding='utf-8')
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""异步上下文管理器出口"""
if self.file_handle:
await self.file_handle.close()
def __aiter__(self) -> AsyncIterator[str]:
"""返回异步迭代器对象"""
return self
async def __anext__(self) -> str:
"""异步读取下一行"""
if not self.file_handle:
raise StopAsyncIteration
line = await self.file_handle.readline()
if not line:
raise StopAsyncIteration
return line.strip()
# GLM智能：异步API响应迭代器
class AsyncAPIIterator:
"""异步API响应迭代器"""
def __init__(self, base_url: str, max_pages: int = 5):
self.base_url = base_url
self.max_pages = max_pages
self.current_page = 1
self.session = None
async def __aenter__(self):
"""创建HTTP会话"""
self.session = aiohttp.ClientSession()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""关闭HTTP会话"""
if self.session:
await self.session.close()
def __aiter__(self) -> AsyncIterator[List[Dict[str, Any]]]:
"""返回异步迭代器对象"""
return self
async def __anext__(self) -> List[Dict[str, Any]]:
"""异步获取下一页数据"""
if self.current_page > self.max_pages:
raise StopAsyncIteration
# 模拟API请求
await asyncio.sleep(0.5) # 模拟网络延迟
# 模拟API响应数据
mock_data = []
for i in range(10): # 每页10条数据
item_id = (self.current_page - 1) * 10 + i + 1
mock_data.append({
'id': item_id,
'title': f'Item {item_id}',
'page': self.current_page,
'timestamp': datetime.now().isoformat()
})
print(f"获取第 {self.current_page} 页数据: {len(mock_data)} 条记录")
self.current_page += 1
return mock_data
# GLM智能：异步数据库查询迭代器
class AsyncDatabaseIterator:
"""异步数据库查询迭代器"""
def __init__(self, query: str, batch_size: int = 50):
self.query = query
self.batch_size = batch_size
self.offset = 0
self.connection = None
self.total_processed = 0
async def __aenter__(self):
"""建立数据库连接（模拟）"""
print("建立数据库连接...")
await asyncio.sleep(0.2) # 模拟连接时间
self.connection = "mock_connection"
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""关闭数据库连接（模拟）"""
if self.connection:
print("关闭数据库连接...")
await asyncio.sleep(0.1) # 模拟关闭时间
self.connection = None
def __aiter__(self) -> AsyncIterator[List[Dict[str, Any]]]:
"""返回异步迭代器对象"""
return self
async def __anext__(self) -> List[Dict[str, Any]]:
"""异步获取下一批数据"""
await asyncio.sleep(0.1) # 模拟查询延迟
# 模拟数据库查询结果
if self.offset >= 200: # 假设总共200条记录
raise StopAsyncIteration
batch = []
remaining = self.batch_size
if self.offset + remaining > 200:
remaining = 200 - self.offset
for i in range(remaining):
record = {
'id': self.offset + i + 1,
'name': f'User_{self.offset + i + 1}',
'age': random.randint(18, 65),
'email': f'user{self.offset + i + 1}@example.com',
'batch': self.total_processed // self.batch_size + 1
}
batch.append(record)
print(f"查询批次 {self.total_processed // self.batch_size + 1}: {len(batch)} 条记录")
self.offset += len(batch)
self.total_processed += len(batch)
return batch
# GLM智能：异步多源数据聚合迭代器
class AsyncMultiSourceIterator:
"""异步多源数据聚合迭代器"""
def __init__(self, sources: List[str]):
self.sources = sources
self.current_source_index = 0
self.current_iterator = None
self.source_name = None
def __aiter__(self) -> AsyncIterator[Dict[str, Any]]:
"""返回异步迭代器对象"""
return self
async def __anext__(self) -> Dict[str, Any]:
"""异步获取下一个数据项"""
while self.current_source_index < len(self.sources):
# 如果当前没有迭代器，创建新的
if self.current_iterator is None:
self.source_name = self.sources[self.current_source_index]
self.current_iterator = self._create_source_iterator(self.source_name)
try:
# 从当前迭代器获取数据
item = await self.current_iterator.__anext__()
item['source'] = self.source_name
return item
except StopAsyncIteration:
# 当前迭代器耗尽，移动到下一个源
self.current_iterator = None
self.current_source_index += 1
raise StopAsyncIteration
def _create_source_iterator(self, source_name: str):
"""为指定源创建迭代器"""
if source_name == "file":
return AsyncFileIterator("mock_data.txt")
elif source_name == "api":
return MockAPIIterator()
elif source_name == "database":
return AsyncDatabaseIterator("SELECT * FROM table", 20)
else:
return AsyncCounter(0, 10, 0.1)
# GLM智能：模拟文件迭代器
class AsyncFileIterator:
"""模拟异步文件迭代器"""
def __init__(self, file_path: str):
self.file_path = file_path
self.lines = [
f"文件行 {i + 1}: 这是一些示例数据内容"
for i in range(15)
]
self.current_index = 0
def __aiter__(self):
return self
async def __anext__(self):
if self.current_index >= len(self.lines):
raise StopAsyncIteration
await asyncio.sleep(0.02)
line = self.lines[self.current_index]
self.current_index += 1
return {'content': line, 'line_number': self.current_index}
# GLM智能：模拟API迭代器
class MockAPIIterator:
"""模拟API迭代器"""
def __init__(self):
self.items = [
{'id': i, 'data': f'API数据 {i}', 'type': 'api_response'}
for i in range(1, 13)
]
self.current_index = 0
def __aiter__(self):
return self
async def __anext__(self):
if self.current_index >= len(self.items):
raise StopAsyncIteration
await asyncio.sleep(0.05)
item = self.items[self.current_index]
self.current_index += 1
return item
# GLM智能：基础异步迭代器演示
async def basic_async_iterator_demo():
"""基础异步迭代器演示"""
print("=== 基础异步迭代器演示 ===")
# 使用自定义异步计数器
counter = AsyncCounter(0, 8, 0.15)
print("使用 async for 遍历异步计数器:")
async for number in counter:
print(f" 计数值: {number}")
print("基础迭代器演示完成\n")
# GLM智能：异步上下文管理迭代器演示
async def context_manager_demo():
"""异步上下文管理迭代器演示"""
print("=== 异步上下文管理迭代器演示 ===")
# 创建临时文件用于演示
temp_file = "demo_async_file.txt"
async with aiofiles.open(temp_file, mode='w', encoding='utf-8') as f:
await f.write("第一行数据\n")
await f.write("第二行数据\n")
await f.write("第三行数据\n")
await f.write("第四行数据\n")
await f.write("第五行数据\n")
# 使用异步文件迭代器
async with AsyncFileReader(temp_file) as file_reader:
print("逐行读取文件内容:")
line_number = 1
async for line in file_reader:
print(f" 第{line_number}行: {line}")
line_number += 1
print("文件迭代器演示完成\n")
# GLM智能：异步API迭代器演示
async def api_iterator_demo():
"""异步API迭代器演示"""
print("=== 异步API迭代器演示 ===")
async with AsyncAPIIterator("https://api.example.com/data", 3) as api_iter:
print("分页获取API数据:")
page_number = 1
async for page_data in api_iter:
print(f" 第{page_number}页包含 {len(page_data)} 条记录")
# 显示前几条记录
for item in page_data[:2]:
print(f" - {item}")
if len(page_data) > 2:
print(f" - ... 还有 {len(page_data) - 2} 条记录")
page_number += 1
print("API迭代器演示完成\n")
# GLM智能：异步数据库迭代器演示
async def database_iterator_demo():
"""异步数据库迭代器演示"""
print("=== 异步数据库迭代器演示 ===")
async with AsyncDatabaseIterator("SELECT * FROM users", 30) as db_iter:
print("批量查询数据库记录:")
batch_number = 1
total_records = 0
async for batch in db_iter:
print(f" 批次{batch_number}: {len(batch)} 条记录")
total_records += len(batch)
# 显示一些样本记录
if batch:
sample = batch[0]
print(f" 样本记录: {sample}")
batch_number += 1
print(f"总共查询了 {total_records} 条记录")
print("数据库迭代器演示完成\n")
# GLM智能：多源聚合迭代器演示
async def multi_source_demo():
"""多源聚合迭代器演示"""
print("=== 多源聚合迭代器演示 ===")
sources = ["file", "api", "database"]
multi_iter = AsyncMultiSourceIterator(sources)
print("聚合多个数据源:")
item_count = 0
source_counts = {}
async for item in multi_iter:
source = item['source']
if source not in source_counts:
source_counts[source] = 0
source_counts[source] += 1
item_count += 1
print(f" 项目 {item_count}: 来源={source}, 数据={item}")
# 限制输出数量
if item_count >= 20:
print(" ... (达到显示限制)")
break
print(f"各源统计: {source_counts}")
print("多源聚合演示完成\n")
# GLM智能：异步迭代器链式处理演示
async def iterator_chain_demo():
"""异步迭代器链式处理演示"""
print("=== 异步迭代器链式处理演示 ===")
# 创建数据源
numbers = AsyncCounter(1, 11, 0.05)
# 第一层：过滤偶数
class EvenFilter:
def __init__(self, source):
self.source = source
def __aiter__(self):
return self
async def __anext__(self):
while True:
value = await self.source.__anext__()
if value % 2 == 0:
return value
# 第二层：平方转换
class SquareTransform:
def __init__(self, source):
self.source = source
def __aiter__(self):
return self
async def __anext__(self):
value = await self.source.__anext__()
return value ** 2
# 构建处理链
even_filter = EvenFilter(numbers)
square_transform = SquareTransform(even_filter)
print("处理链: 数字 -> 过滤偶数 -> 平方变换")
async for result in square_transform:
print(f" 处理结果: {result}")
print("链式处理演示完成\n")
# GLM智能：异步迭代器性能演示
async def performance_demo():
"""异步迭代器性能演示"""
print("=== 异步迭代器性能演示 ===")
# 大量数据异步处理
large_data = AsyncCounter(0, 1000, 0.001) # 1000个项目
print("处理1000个数据项:")
start_time = datetime.now()
processed_count = 0
async for item in large_data:
processed_count += 1
# 简单处理
_ = item * 2
if processed_count % 100 == 0:
print(f" 已处理: {processed_count} 项")
end_time = datetime.now()
duration = (end_time - start_time).total_seconds()
print(f"处理完成: {processed_count} 项，耗时: {duration:.2f}秒")
print(f"平均每项处理时间: {duration/processed_count*1000:.2f}毫秒\n")
# GLM智能：异步迭代器错误处理演示
async def error_handling_demo():
"""异步迭代器错误处理演示"""
print("=== 异步迭代器错误处理演示 ===")
class FaultyIterator:
def __init__(self, fail_at: int = 3):
self.current = 0
self.fail_at = fail_at
def __aiter__(self):
return self
async def __anext__(self):
await asyncio.sleep(0.1)
self.current += 1
if self.current == self.fail_at:
raise ValueError(f"模拟错误在第 {self.current} 次迭代时发生")
if self.current > 6:
raise StopAsyncIteration
return self.current
faulty_iter = FaultyIterator(4)
try:
print("使用可能出错的迭代器:")
async for item in faulty_iter:
print(f" 获取项目: {item}")
except ValueError as e:
print(f"捕获到预期错误: {e}")
except Exception as e:
print(f"未预期错误: {e}")
print("错误处理演示完成\n")
# GLM智能：主演示函数
async def main():
"""主演示函数"""
print("=== 异步迭代器完整演示 ===\n")
# 1. 基础异步迭代器演示
await basic_async_iterator_demo()
# 2. 异步上下文管理迭代器演示
await context_manager_demo()
# 3. 异步API迭代器演示
await api_iterator_demo()
# 4. 异步数据库迭代器演示
await database_iterator_demo()
# 5. 多源聚合迭代器演示
await multi_source_demo()
# 6. 异步迭代器链式处理演示
await iterator_chain_demo()
# 7. 异步迭代器性能演示
await performance_demo()
# 8. 异步迭代器错误处理演示
await error_handling_demo()
print("=== 异步迭代器演示完成 ===")
# 运行演示
if __name__ == "__main__":
asyncio.run(main())
---
8. 并发模式
8.1 生产者消费者模式
01.生产者消费者模式概述
a.基本介绍
生产者消费者模式是一种经典的并发设计模式，用于解决生产者和消费者之间的数据交换问题。
在该模式中，生产者负责生产数据并放入缓冲区，消费者从缓冲区中取出数据并进行处理，两者通过缓冲区实现解耦。
这种模式能够有效平衡生产者和消费者的处理速度差异，避免因速度不匹配导致的资源浪费或性能下降问题。
Python中可以使用queue.Queue、multiprocessing.Queue或asyncio.Queue等数据结构来实现生产者消费者模式。
b.核心特性
a.解耦合
生产者和消费者相互独立，互不直接依赖，通过缓冲区进行通信，降低了系统的耦合度。
b.并发处理
生产者和消费者可以并发执行，充分利用多核处理器的优势，提高系统的整体处理能力。
c.缓冲机制
通过缓冲区平衡生产和消费速度，当生产速度大于消费速度时，数据暂存于缓冲区中。
d.线程安全
内置的队列数据结构提供线程安全保证，避免了手动实现同步机制的复杂性。
c.代码示例
---
# GLM智能：生产者消费者模式完整示例
import threading
import queue
import time
import random
from typing import List, Dict, Any, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime
from enum import Enum
# GLM智能：数据类型定义
class TaskType(Enum):
"""任务类型枚举"""
DATA_PROCESSING = "data_processing"
FILE_ANALYSIS = "file_analysis"
NETWORK_REQUEST = "network_request"
CALCULATION = "calculation"
@dataclass
class Task:
"""任务数据类"""
task_id: int
task_type: TaskType
data: Dict[str, Any]
priority: int = 1
created_at: datetime = None
def __post_init__(self):
if self.created_at is None:
self.created_at = datetime.now()
@dataclass
class Result:
"""结果数据类"""
task_id: int
result: Any
processing_time: float
worker_id: str
completed_at: datetime = None
def __post_init__(self):
if self.completed_at is None:
self.completed_at = datetime.now()
# GLM智能：基础生产者消费者实现
class BasicProducerConsumer:
"""基础生产者消费者实现"""
def __init__(self, max_queue_size: int = 10):
self.task_queue = queue.Queue(maxsize=max_queue_size)
self.result_queue = queue.Queue()
self.producers = []
self.consumers = []
self.is_running = False
self.task_counter = 0
def add_producer(self, producer_id: str, production_rate: float = 1.0):
"""添加生产者"""
def producer_worker():
while self.is_running:
try:
# 生成任务
task = self._create_task(producer_id)
# 将任务放入队列
self.task_queue.put(task)
print(f"生产者 {producer_id} 生产任务 {task.task_id}")
# 控制生产速度
time.sleep(1.0 / production_rate)
except Exception as e:
print(f"生产者 {producer_id} 出错: {e}")
producer_thread = threading.Thread(target=producer_worker, name=f"Producer-{producer_id}")
self.producers.append(producer_thread)
def add_consumer(self, consumer_id: str, processing_speed: float = 1.0):
"""添加消费者"""
def consumer_worker():
while self.is_running or not self.task_queue.empty():
try:
# 从队列获取任务（超时机制防止死锁）
task = self.task_queue.get(timeout=1.0)
# 处理任务
start_time = time.time()
result_data = self._process_task(task, consumer_id)
processing_time = time.time() - start_time
# 创建结果对象
result = Result(
task_id=task.task_id,
result=result_data,
processing_time=processing_time,
worker_id=consumer_id
)
# 将结果放入结果队列
self.result_queue.put(result)
print(f"消费者 {consumer_id} 完成任务 {task.task_id}，耗时 {processing_time:.2f}秒")
# 标记任务完成
self.task_queue.task_done()
except queue.Empty:
continue
except Exception as e:
print(f"消费者 {consumer_id} 出错: {e}")
consumer_thread = threading.Thread(target=consumer_worker, name=f"Consumer-{consumer_id}")
self.consumers.append(consumer_thread)
def start(self):
"""启动生产者消费者系统"""
self.is_running = True
# 启动所有生产者
for producer in self.producers:
producer.start()
print(f"启动生产者: {producer.name}")
# 启动所有消费者
for consumer in self.consumers:
consumer.start()
print(f"启动消费者: {consumer.name}")
def stop(self):
"""停止生产者消费者系统"""
self.is_running = False
# 等待所有生产者结束
for producer in self.producers:
producer.join(timeout=2.0)
print(f"生产者 {producer.name} 已停止")
# 等待所有消费者结束
for consumer in self.consumers:
consumer.join(timeout=2.0)
print(f"消费者 {consumer.name} 已停止")
def get_results(self) -> List[Result]:
"""获取所有结果"""
results = []
while not self.result_queue.empty():
try:
result = self.result_queue.get_nowait()
results.append(result)
except queue.Empty:
break
return results
def _create_task(self, producer_id: str) -> Task:
"""创建新任务"""
self.task_counter += 1
task_type = random.choice(list(TaskType))
data = {
'producer_id': producer_id,
'payload_size': random.randint(100, 1000),
'complexity': random.randint(1, 5)
}
return Task(
task_id=self.task_counter,
task_type=task_type,
data=data,
priority=random.randint(1, 3)
)
def _process_task(self, task: Task, consumer_id: str) -> Dict[str, Any]:
"""处理任务"""
# 模拟不同类型任务的处理
processing_time = task.data['complexity'] * 0.1
time.sleep(processing_time)
return {
'consumer_id': consumer_id,
'original_data': task.data,
'processed_size': task.data['payload_size'] * 2,
'processing_method': f"process_{task.task_type.value}",
'success': True
}
# GLM智能：优先级队列生产者消费者
class PriorityProducerConsumer:
"""基于优先级的生产者消费者实现"""
def __init__(self, max_queue_size: int = 10):
self.task_queue = queue.PriorityQueue(maxsize=max_queue_size)
self.result_queue = queue.Queue()
self.producers = []
self.consumers = []
self.is_running = False
self.task_counter = 0
def add_producer(self, producer_id: str, production_rate: float = 1.0):
"""添加生产者"""
def producer_worker():
while self.is_running:
try:
# 生成带优先级的任务
task = self._create_priority_task(producer_id)
# 将任务放入优先级队列 (priority, task)
priority_item = (-task.priority, task) # 负值实现高优先级先处理
self.task_queue.put(priority_item)
print(f"生产者 {producer_id} 生产优先级 {task.priority} 的任务 {task.task_id}")
time.sleep(1.0 / production_rate)
except Exception as e:
print(f"生产者 {producer_id} 出错: {e}")
producer_thread = threading.Thread(target=producer_worker, name=f"PriorityProducer-{producer_id}")
self.producers.append(producer_thread)
def add_consumer(self, consumer_id: str, processing_speed: float = 1.0):
"""添加消费者"""
def consumer_worker():
while self.is_running or not self.task_queue.empty():
try:
# 从优先级队列获取任务
priority, task = self.task_queue.get(timeout=1.0)
# 处理任务
start_time = time.time()
result_data = self._process_task(task, consumer_id)
processing_time = time.time() - start_time
# 创建结果对象
result = Result(
task_id=task.task_id,
result=result_data,
processing_time=processing_time,
worker_id=consumer_id
)
self.result_queue.put(result)
print(f"消费者 {consumer_id} 完成优先级 {-priority} 的任务 {task.task_id}")
self.task_queue.task_done()
except queue.Empty:
continue
except Exception as e:
print(f"消费者 {consumer_id} 出错: {e}")
consumer_thread = threading.Thread(target=consumer_worker, name=f"PriorityConsumer-{consumer_id}")
self.consumers.append(consumer_thread)
def start(self):
"""启动系统"""
self.is_running = True
for producer in self.producers:
producer.start()
for consumer in self.consumers:
consumer.start()
def stop(self):
"""停止系统"""
self.is_running = False
for producer in self.producers:
producer.join(timeout=2.0)
for consumer in self.consumers:
consumer.join(timeout=2.0)
def _create_priority_task(self, producer_id: str) -> Task:
"""创建带优先级的任务"""
self.task_counter += 1
task_type = random.choice(list(TaskType))
data = {
'producer_id': producer_id,
'priority_level': random.randint(1, 5),
'deadline': time.time() + random.randint(5, 30)
}
# 创建基于数据优先级的任务
priority = data['priority_level']
return Task(
task_id=self.task_counter,
task_type=task_type,
data=data,
priority=priority
)
def _process_task(self, task: Task, consumer_id: str) -> Dict[str, Any]:
"""处理任务"""
processing_time = task.data['priority_level'] * 0.05
time.sleep(processing_time)
return {
'consumer_id': consumer_id,
'original_data': task.data,
'priority_handled': task.priority,
'processing_method': f"priority_process_{task.task_type.value}",
'success': True
}
# GLM智能：批量处理生产者消费者
class BatchProducerConsumer:
"""支持批量处理的生产者消费者"""
def __init__(self, max_queue_size: int = 10, batch_size: int = 5):
self.task_queue = queue.Queue(maxsize=max_queue_size)
self.result_queue = queue.Queue()
self.producers = []
self.consumers = []
self.is_running = False
self.task_counter = 0
self.batch_size = batch_size
def add_batch_producer(self, producer_id: str, batch_production_rate: float = 0.5):
"""添加批量生产者"""
def batch_producer_worker():
while self.is_running:
try:
# 生成一批任务
batch_tasks = self._create_task_batch(producer_id)
# 将批量任务放入队列
self.task_queue.put(batch_tasks)
print(f"批量生产者 {producer_id} 生产 {len(batch_tasks)} 个任务")
time.sleep(1.0 / batch_production_rate)
except Exception as e:
print(f"批量生产者 {producer_id} 出错: {e}")
producer_thread = threading.Thread(target=batch_producer_worker, name=f"BatchProducer-{producer_id}")
self.producers.append(producer_thread)
def add_batch_consumer(self, consumer_id: str, batch_processing_speed: float = 1.0):
"""添加批量消费者"""
def batch_consumer_worker():
while self.is_running or not self.task_queue.empty():
try:
# 获取批量任务
batch_tasks = self.task_queue.get(timeout=1.0)
# 批量处理任务
start_time = time.time()
batch_results = self._process_task_batch(batch_tasks, consumer_id)
processing_time = time.time() - start_time
# 将批量结果放入结果队列
self.result_queue.put(batch_results)
print(f"批量消费者 {consumer_id} 处理 {len(batch_tasks)} 个任务，耗时 {processing_time:.2f}秒")
self.task_queue.task_done()
except queue.Empty:
continue
except Exception as e:
print(f"批量消费者 {consumer_id} 出错: {e}")
consumer_thread = threading.Thread(target=batch_consumer_worker, name=f"BatchConsumer-{consumer_id}")
self.consumers.append(consumer_thread)
def start(self):
"""启动系统"""
self.is_running = True
for producer in self.producers:
producer.start()
for consumer in self.consumers:
consumer.start()
def stop(self):
"""停止系统"""
self.is_running = False
for producer in self.producers:
producer.join(timeout=2.0)
for consumer in self.consumers:
consumer.join(timeout=2.0)
def _create_task_batch(self, producer_id: str) -> List[Task]:
"""创建任务批次"""
batch_tasks = []
for _ in range(self.batch_size):
self.task_counter += 1
task_type = random.choice(list(TaskType))
data = {
'producer_id': producer_id,
'batch_id': self.task_counter // self.batch_size,
'size': random.randint(50, 200)
}
task = Task(
task_id=self.task_counter,
task_type=task_type,
data=data
)
batch_tasks.append(task)
return batch_tasks
def _process_task_batch(self, batch_tasks: List[Task], consumer_id: str) -> List[Result]:
"""批量处理任务"""
batch_results = []
for task in batch_tasks:
processing_time = 0.1 # 批量处理单个任务时间更短
time.sleep(processing_time)
result_data = {
'consumer_id': consumer_id,
'original_data': task.data,
'batch_processed': True,
'processing_method': f"batch_process_{task.task_type.value}",
'success': True
}
result = Result(
task_id=task.task_id,
result=result_data,
processing_time=processing_time,
worker_id=consumer_id
)
batch_results.append(result)
return batch_results
# GLM智能：基础演示
def basic_producer_consumer_demo():
"""基础生产者消费者演示"""
print("=== 基础生产者消费者演示 ===")
# 创建生产者消费者系统
pc_system = BasicProducerConsumer(max_queue_size=5)
# 添加2个生产者
pc_system.add_producer("Producer-A", production_rate=2.0)
pc_system.add_producer("Producer-B", production_rate=1.5)
# 添加3个消费者
pc_system.add_consumer("Consumer-X", processing_speed=1.0)
pc_system.add_consumer("Consumer-Y", processing_speed=1.2)
pc_system.add_consumer("Consumer-Z", processing_speed=0.8)
# 启动系统
pc_system.start()
# 运行一段时间
time.sleep(10)
# 停止系统
pc_system.stop()
# 获取结果
results = pc_system.get_results()
print(f"\n总处理任务数: {len(results)}")
# 统计每个消费者处理的任务数
consumer_stats = {}
for result in results:
consumer_id = result.worker_id
consumer_stats[consumer_id] = consumer_stats.get(consumer_id, 0) + 1
for consumer_id, count in consumer_stats.items():
print(f"{consumer_id} 处理了 {count} 个任务")
# GLM智能：优先级队列演示
def priority_producer_consumer_demo():
"""优先级队列生产者消费者演示"""
print("\n=== 优先级队列生产者消费者演示 ===")
# 创建优先级生产者消费者系统
priority_system = PriorityProducerConsumer(max_queue_size=8)
# 添加生产者
priority_system.add_producer("HighPriorityProducer", production_rate=1.5)
priority_system.add_producer("LowPriorityProducer", production_rate=1.0)
# 添加消费者
priority_system.add_consumer("FastConsumer", processing_speed=1.5)
priority_system.add_consumer("NormalConsumer", processing_speed=1.0)
# 启动系统
priority_system.start()
# 运行一段时间
time.sleep(8)
# 停止系统
priority_system.stop()
# GLM智能：批量处理演示
def batch_producer_consumer_demo():
"""批量处理生产者消费者演示"""
print("\n=== 批量处理生产者消费者演示 ===")
# 创建批量处理系统
batch_system = BatchProducerConsumer(max_queue_size=6, batch_size=4)
# 添加批量生产者
batch_system.add_batch_producer("BatchProducer-A", batch_production_rate=1.0)
batch_system.add_batch_producer("BatchProducer-B", batch_production_rate=0.8)
# 添加批量消费者
batch_system.add_batch_consumer("BatchConsumer-X", batch_processing_speed=1.2)
batch_system.add_batch_consumer("BatchConsumer-Y", batch_processing_speed=1.0)
# 启动系统
batch_system.start()
# 运行一段时间
time.sleep(10)
# 停止系统
batch_system.stop()
# GLM智能：主演示函数
def main():
"""主演示函数"""
print("=== 生产者消费者模式完整演示 ===\n")
# 1. 基础生产者消费者演示
basic_producer_consumer_demo()
# 2. 优先级队列演示
priority_producer_consumer_demo()
# 3. 批量处理演示
batch_producer_consumer_demo()
print("\n=== 生产者消费者模式演示完成 ===")
# 运行演示
if __name__ == "__main__":
main()
---
8.2 读写锁模式
01.读写锁模式概述
a.基本介绍
读写锁模式是一种允许多个线程同时读取共享数据，但只允许一个线程写入数据的同步机制。
这种模式基于"读多写少"的场景优化性能，区分读操作和写操作，提供不同的访问权限控制。
读写锁的基本原理是：多个读操作可以并发执行，但写操作是排他的，需要独占访问，读写操作不能同时进行。
Python中可以使用threading模块结合条件变量或第三方库来实现读写锁模式，平衡数据一致性和并发性能。
b.核心特性
a.读操作共享
多个线程可以同时持有读锁并访问共享资源，提高了读密集型应用的并发性能。
b.写操作排他
写操作需要独占锁，确保在写入期间没有其他线程进行读写操作，保证数据一致性。
c.锁升级机制
某些实现支持从读锁升级到写锁，但需要小心处理避免死锁问题。
d.公平性策略
可以实现公平或非公平的锁获取策略，平衡吞吐量和线程等待时间。
c.代码示例
---
# GLM智能：读写锁模式完整示例
import threading
import time
import random
from typing import Dict, List, Any, Optional, Callable
from dataclasses import dataclass
from datetime import datetime
from enum import Enum
from contextlib import contextmanager
collections = __import__('collections')
# GLM智能：操作类型定义
class OperationType(Enum):
"""操作类型枚举"""
READ = "read"
WRITE = "write"
@dataclass
class Operation:
"""操作记录数据类"""
operation_id: int
operation_type: OperationType
thread_id: str
start_time: float
end_time: Optional[float] = None
success: bool = False
# GLM智能：基础读写锁实现
class ReadWriteLock:
"""基础读写锁实现"""
def __init__(self):
self._read_ready = threading.Condition(threading.RLock())
self._readers = 0 # 当前读线程数量
def acquire_read(self):
"""获取读锁"""
with self._read_ready:
self._readers += 1
def release_read(self):
"""释放读锁"""
with self._read_ready:
self._readers -= 1
if self._readers == 0:
# 如果没有读者了，通知等待的写者
self._read_ready.notifyAll()
def acquire_write(self):
"""获取写锁"""
self._read_ready.acquire()
while self._readers > 0:
# 等待所有读者完成
self._read_ready.wait()
def release_write(self):
"""释放写锁"""
self._read_ready.release()
@contextmanager
def read_lock(self):
"""读锁上下文管理器"""
self.acquire_read()
try:
yield
finally:
self.release_read()
@contextmanager
def write_lock(self):
"""写锁上下文管理器"""
self.acquire_write()
try:
yield
finally:
self.release_write()
# GLM智能：可重入读写锁实现
class ReentrantReadWriteLock:
"""可重入读写锁实现"""
def __init__(self):
self._read_ready = threading.Condition(threading.RLock())
self._readers = 0 # 当前读线程数量
self._writers = 0 # 当前写线程数量
self._writing = False # 是否正在写入
self._readers_count = {} # 每个线程的重入读锁计数
self._writer_lock = threading.Lock() # 写者锁
def acquire_read(self):
"""获取可重入读锁"""
current_thread = threading.current_thread()
self._read_ready.acquire()
try:
# 如果当前线程已经持有读锁，增加计数
if current_thread in self._readers_count:
self._readers_count[current_thread] += 1
self._readers += 1
return
# 等待写操作完成
while self._writing or self._writers > 0:
self._read_ready.wait()
# 首次获取读锁
self._readers_count[current_thread] = 1
self._readers += 1
finally:
self._read_ready.release()
def release_read(self):
"""释放可重入读锁"""
current_thread = threading.current_thread()
self._read_ready.acquire()
try:
if current_thread not in self._readers_count:
raise RuntimeError("当前线程未持有读锁")
self._readers_count[current_thread] -= 1
self._readers -= 1
if self._readers_count[current_thread] == 0:
del self._readers_count[current_thread]
# 如果没有读者了，通知等待的写者
if self._readers == 0:
self._read_ready.notifyAll()
finally:
self._read_ready.release()
def acquire_write(self):
"""获取可重入写锁"""
current_thread = threading.current_thread()
with self._writer_lock:
# 如果当前线程已经持有写锁，允许重入
if hasattr(self, '_current_writer') and self._current_writer == current_thread:
self._writers += 1
return
# 等待获取写锁
self._read_ready.acquire()
self._writers += 1
try:
while self._writing or self._readers > 0:
self._read_ready.wait()
self._writing = True
self._current_writer = current_thread
except:
self._writers -= 1
self._read_ready.release()
raise
def release_write(self):
"""释放可重入写锁"""
current_thread = threading.current_thread()
with self._writer_lock:
if not hasattr(self, '_current_writer') or self._current_writer != current_thread:
raise RuntimeError("当前线程未持有写锁")
self._writers -= 1
if self._writers == 0:
self._writing = False
delattr(self, '_current_writer')
self._read_ready.notifyAll()
self._read_ready.release()
# GLM智能：公平读写锁实现
class FairReadWriteLock:
"""公平读写锁实现"""
def __init__(self):
self._lock = threading.Lock()
self._readers = 0
self._writers = 0
self._write_pending = 0
self._read_condition = threading.Condition(self._lock)
self._write_condition = threading.Condition(self._lock)
def acquire_read(self):
"""获取公平读锁"""
with self._lock:
# 等待写操作完成
while self._writers > 0 or self._write_pending > 0:
self._read_condition.wait()
self._readers += 1
def release_read(self):
"""释放公平读锁"""
with self._lock:
self._readers -= 1
if self._readers == 0:
# 通知等待的写者
self._write_condition.notifyAll()
def acquire_write(self):
"""获取公平写锁"""
with self._lock:
self._write_pending += 1
try:
# 等待所有读者和写者完成
while self._readers > 0 or self._writers > 0:
self._write_condition.wait()
self._writers += 1
except:
self._write_pending -= 1
raise
def release_write(self):
"""释放公平写锁"""
with self._lock:
self._writers -= 1
self._write_pending -= 1
# 优先通知等待的写者
if self._write_pending > 0:
self._write_condition.notify()
else:
# 没有等待的写者时通知读者
self._read_condition.notifyAll()
# GLM智能：读写锁数据存储
class ReadWriteLockDataStore:
"""使用读写锁保护的数据存储"""
def __init__(self, lock_type: str = "basic"):
if lock_type == "basic":
self.rw_lock = ReadWriteLock()
elif lock_type == "reentrant":
self.rw_lock = ReentrantReadWriteLock()
elif lock_type == "fair":
self.rw_lock = FairReadWriteLock()
else:
raise ValueError(f"不支持的锁类型: {lock_type}")
self._data = {}
self._operation_log = []
self._operation_counter = 0
def read_data(self, key: str, reader_id: str) -> Any:
"""读取数据"""
with self.rw_lock.read_lock():
self._log_operation(OperationType.READ, reader_id)
time.sleep(0.01) # 模拟读取延迟
return self._data.get(key, "默认值")
def write_data(self, key: str, value: Any, writer_id: str) -> bool:
"""写入数据"""
with self.rw_lock.write_lock():
self._log_operation(OperationType.WRITE, writer_id)
time.sleep(0.05) # 模拟写入延迟
self._data[key] = value
return True
def read_all_data(self, reader_id: str) -> Dict[str, Any]:
"""读取所有数据"""
with self.rw_lock.read_lock():
self._log_operation(OperationType.READ, reader_id)
time.sleep(0.02) # 模拟读取延迟
return self._data.copy()
def batch_write_data(self, items: Dict[str, Any], writer_id: str) -> int:
"""批量写入数据"""
with self.rw_lock.write_lock():
self._log_operation(OperationType.WRITE, writer_id)
time.sleep(len(items) * 0.02) # 模拟批量写入延迟
self._data.update(items)
return len(items)
def conditional_update(self, key: str, old_value: Any, new_value: Any, writer_id: str) -> bool:
"""条件更新（需要读锁和写锁）"""
# 先检查当前值（读锁）
with self.rw_lock.read_lock():
current_value = self._data.get(key)
if current_value != old_value:
return False
# 再更新值（写锁）
with self.rw_lock.write_lock():
self._log_operation(OperationType.WRITE, writer_id)
# 再次检查，防止在获得写锁之前数据被修改
if self._data.get(key) != old_value:
return False
self._data[key] = new_value
return True
def get_operation_statistics(self) -> Dict[str, Any]:
"""获取操作统计信息"""
with self.rw_lock.read_lock():
read_ops = [op for op in self._operation_log if op.operation_type == OperationType.READ]
write_ops = [op for op in self._operation_log if op.operation_type == OperationType.WRITE]
return {
'total_operations': len(self._operation_log),
'read_operations': len(read_ops),
'write_operations': len(write_ops),
'data_size': len(self._data),
'average_read_time': self._calculate_avg_time(read_ops),
'average_write_time': self._calculate_avg_time(write_ops)
}
def _log_operation(self, op_type: OperationType, thread_id: str):
"""记录操作"""
self._operation_counter += 1
operation = Operation(
operation_id=self._operation_counter,
operation_type=op_type,
thread_id=thread_id,
start_time=time.time()
)
self._operation_log.append(operation)
return operation
def _calculate_avg_time(self, operations: List[Operation]) -> float:
"""计算平均操作时间"""
if not operations:
return 0.0
total_time = sum(op.end_time - op.start_time for op in operations if op.end_time)
count = sum(1 for op in operations if op.end_time)
return total_time / count if count > 0 else 0.0
# GLM智能：缓存系统实现
class ReadWriteLockCache:
"""使用读写锁的缓存系统"""
def __init__(self, max_size: int = 100):
self.rw_lock = ReadWriteLock()
self.cache = {}
self.access_order = collections.deque(maxlen=max_size)
self.max_size = max_size
self.hits = 0
self.misses = 0
def get(self, key: str) -> Optional[Any]:
"""获取缓存值"""
with self.rw_lock.read_lock():
if key in self.cache:
self.hits += 1
# 更新访问顺序（需要写操作，所以先释放读锁）
value = self.cache[key]
else:
self.misses += 1
return None
# 更新访问顺序
with self.rw_lock.write_lock():
if key in self.cache: # 再次检查，防止值被删除
self.access_order.remove(key)
self.access_order.append(key)
return self.cache[key]
return value
def put(self, key: str, value: Any):
"""设置缓存值"""
with self.rw_lock.write_lock():
if len(self.cache) >= self.max_size and key not in self.cache:
# LRU淘汰
oldest_key = self.access_order.popleft()
del self.cache[oldest_key]
self.cache[key] = value
if key in self.access_order:
self.access_order.remove(key)
self.access_order.append(key)
def clear(self):
"""清空缓存"""
with self.rw_lock.write_lock():
self.cache.clear()
self.access_order.clear()
def get_statistics(self) -> Dict[str, Any]:
"""获取缓存统计"""
with self.rw_lock.read_lock():
total_requests = self.hits + self.misses
hit_rate = self.hits / total_requests if total_requests > 0 else 0
return {
'cache_size': len(self.cache),
'max_size': self.max_size,
'hits': self.hits,
'misses': self.misses,
'hit_rate': hit_rate
}
# GLM智能：读写操作线程
class ReaderThread(threading.Thread):
"""读者线程"""
def __init__(self, data_store: ReadWriteLockDataStore, reader_id: str,
operations: int = 20, keys: List[str] = None):
super().__init__(name=f"Reader-{reader_id}")
self.data_store = data_store
self.reader_id = reader_id
self.operations = operations
self.keys = keys or ["key1", "key2", "key3", "key4", "key5"]
self.results = []
def run(self):
"""执行读操作"""
for i in range(self.operations):
try:
key = random.choice(self.keys)
start_time = time.time()
# 随机选择读取单个键或全部数据
if random.random() < 0.7: # 70%概率读取单个键
value = self.data_store.read_data(key, self.reader_id)
else: # 30%概率读取全部数据
value = self.data_store.read_all_data(self.reader_id)
end_time = time.time()
self.results.append({
'operation': 'read',
'key': key,
'value': value,
'duration': end_time - start_time,
'timestamp': end_time
})
time.sleep(random.uniform(0.01, 0.05))
except Exception as e:
print(f"读者 {self.reader_id} 出错: {e}")
class WriterThread(threading.Thread):
"""写者线程"""
def __init__(self, data_store: ReadWriteLockDataStore, writer_id: str,
operations: int = 10, keys: List[str] = None):
super().__init__(name=f"Writer-{writer_id}")
self.data_store = data_store
self.writer_id = writer_id
self.operations = operations
self.keys = keys or ["key1", "key2", "key3", "key4", "key5"]
self.results = []
def run(self):
"""执行写操作"""
for i in range(self.operations):
try:
key = random.choice(self.keys)
value = f"{self.writer_id}_value_{i}_{random.randint(1000, 9999)}"
start_time = time.time()
# 随机选择写入单个键或批量写入
if random.random() < 0.8: # 80%概率写入单个键
success = self.data_store.write_data(key, value, self.writer_id)
operation_type = 'single_write'
else: # 20%概率批量写入
batch_items = {
f"batch_key_{j}": f"{self.writer_id}_batch_{i}_{j}"
for j in range(3)
}
count = self.data_store.batch_write_data(batch_items, self.writer_id)
success = count > 0
operation_type = 'batch_write'
key = f"batch_{len(batch_items)}_items"
end_time = time.time()
self.results.append({
'operation': operation_type,
'key': key,
'success': success,
'duration': end_time - start_time,
'timestamp': end_time
})
time.sleep(random.uniform(0.05, 0.15))
except Exception as e:
print(f"写者 {self.writer_id} 出错: {e}")
# GLM智能：基础读写锁演示
def basic_read_write_lock_demo():
"""基础读写锁演示"""
print("=== 基础读写锁演示 ===")
# 创建数据存储
data_store = ReadWriteLockDataStore("basic")
# 预填充一些数据
data_store.write_data("key1", "initial_value_1", "init_writer")
data_store.write_data("key2", "initial_value_2", "init_writer")
# 创建读者和写者线程
readers = [
ReaderThread(data_store, f"Reader-{i}", operations=15)
for i in range(4)
]
writers = [
WriterThread(data_store, f"Writer-{i}", operations=8)
for i in range(2)
]
# 启动所有线程
print("启动读者和写者线程...")
all_threads = readers + writers
for thread in all_threads:
thread.start()
# 等待所有线程完成
for thread in all_threads:
thread.join()
# 显示统计信息
stats = data_store.get_operation_statistics()
print(f"\n操作统计:")
print(f" 总操作数: {stats['total_operations']}")
print(f" 读操作: {stats['read_operations']}")
print(f" 写操作: {stats['write_operations']}")
print(f" 数据项数: {stats['data_size']}")
print(f" 平均读时间: {stats['average_read_time']:.4f}秒")
print(f" 平均写时间: {stats['average_write_time']:.4f}秒")
# GLM智能：可重入读写锁演示
def reentrant_read_write_lock_demo():
"""可重入读写锁演示"""
print("\n=== 可重入读写锁演示 ===")
data_store = ReadWriteLockDataStore("reentrant")
# 演示读锁重入
def nested_read_operations(reader_id: str):
"""嵌套读操作"""
try:
# 第一次读
value1 = data_store.read_data("key1", reader_id)
print(f" {reader_id} 第一次读取: {value1}")
# 嵌套读锁
value2 = data_store.read_data("key2", reader_id)
print(f" {reader_id} 第二次读取: {value2}")
# 再次嵌套
all_data = data_store.read_all_data(reader_id)
print(f" {reader_id} 读取所有数据: {len(all_data)} 项")
except Exception as e:
print(f" {reader_id} 嵌套读操作出错: {e}")
# 创建测试线程
def reentrant_test():
"""可重入测试"""
nested_read_operations("ReentrantReader")
# 运行测试
test_thread = threading.Thread(target=reentrant_test, name="ReentrantTest")
test_thread.start()
test_thread.join()
print("可重入读写锁测试完成")
# GLM智能：公平读写锁演示
def fair_read_write_lock_demo():
"""公平读写锁演示"""
print("\n=== 公平读写锁演示 ===")
data_store = ReadWriteLockDataStore("fair")
# 创建大量读者和少量写者来测试公平性
readers = [
ReaderThread(data_store, f"FairReader-{i}", operations=5)
for i in range(8)
]
writers = [
WriterThread(data_store, f"FairWriter-{i}", operations=3)
for i in range(3)
]
print("启动公平性测试（8个读者，3个写者）...")
all_threads = readers + writers
for thread in all_threads:
thread.start()
# 等待完成
for thread in all_threads:
thread.join()
stats = data_store.get_operation_statistics()
print(f"公平读写锁测试完成:")
print(f" 总操作数: {stats['total_operations']}")
print(f" 读操作: {stats['read_operations']}")
print(f" 写操作: {stats['write_operations']}")
# GLM智能：缓存系统演示
def cache_system_demo():
"""缓存系统演示"""
print("\n=== 读写锁缓存系统演示 ===")
cache = ReadWriteLockCache(max_size=5)
def cache_reader(reader_id: str):
"""缓存读者"""
for i in range(10):
key = f"key_{random.randint(1, 8)}"
value = cache.get(key)
print(f" 缓存读者 {reader_id} 读取 {key}: {value}")
time.sleep(0.01)
def cache_writer(writer_id: str):
"""缓存写者"""
for i in range(5):
key = f"key_{random.randint(1, 8)}"
value = f"value_{writer_id}_{i}"
cache.put(key, value)
print(f" 缓存写者 {writer_id} 写入 {key}: {value}")
time.sleep(0.02)
# 创建缓存操作线程
cache_readers = [
threading.Thread(target=cache_reader, args=(f"CacheReader-{i}",))
for i in range(4)
]
cache_writers = [
threading.Thread(target=cache_writer, args=(f"CacheWriter-{i}",))
for i in range(2)
]
# 启动缓存线程
for thread in cache_readers + cache_writers:
thread.start()
# 等待完成
for thread in cache_readers + cache_writers:
thread.join()
# 显示缓存统计
cache_stats = cache.get_statistics()
print(f"\n缓存统计:")
print(f" 缓存大小: {cache_stats['cache_size']}/{cache_stats['max_size']}")
print(f" 命中次数: {cache_stats['hits']}")
print(f" 未命中次数: {cache_stats['misses']}")
print(f" 命中率: {cache_stats['hit_rate']:.2%}")
# GLM智能：主演示函数
def main():
"""主演示函数"""
print("=== 读写锁模式完整演示 ===\n")
# 1. 基础读写锁演示
basic_read_write_lock_demo()
# 2. 可重入读写锁演示
reentrant_read_write_lock_demo()
# 3. 公平读写锁演示
fair_read_write_lock_demo()
# 4. 缓存系统演示
cache_system_demo()
print("\n=== 读写锁模式演示完成 ===")
# 运行演示
if __name__ == "__main__":
main()
---
8.3 Future模式
01.Future模式概述
a.基本介绍
Future模式是一种并发设计模式，用于表示异步操作的结果，允许在操作完成前继续执行其他任务。
该模式将任务提交与结果获取分离，调用者可以立即获得一个Future对象，后续通过该对象获取实际结果。
Future提供了非阻塞的任务执行方式，有效提高了系统的并发处理能力和资源利用率，特别适合耗时操作的场景。
Python中通过concurrent.futures模块、asyncio.Future或第三方库实现Future模式，支持线程池和进程池的异步任务执行。
b.核心特性
a.异步执行
任务在后台异步执行，调用者无需等待任务完成即可继续执行其他操作。
b.结果代理
Future对象作为任务结果的代理，提供状态查询和结果获取的统一接口。
c.状态管理
提供pending、running、done、cancelled等状态，便于监控任务执行进度。
d.异常处理
支持异步异常传播和处理机制，确保错误信息能够正确传递给调用者。
c.代码示例
---
# GLM智能：Future模式完整示例
import threading
import time
import random
import requests
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed, wait
from typing import List, Dict, Any, Optional, Callable, Union
from dataclasses import dataclass
from datetime import datetime
from enum import Enum
functools = __import__('functools')
# GLM智能：任务状态和类型定义
class TaskStatus(Enum):
"""任务状态枚举"""
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
class TaskType(Enum):
"""任务类型枚举"""
COMPUTATION = "computation"
IO_BOUND = "io_bound"
NETWORK_REQUEST = "network_request"
DATA_PROCESSING = "data_processing"
@dataclass
class TaskResult:
"""任务结果数据类"""
task_id: str
task_type: TaskType
result: Any
execution_time: float
success: bool
error: Optional[str] = None
timestamp: datetime = None
def __post_init__(self):
if self.timestamp is None:
self.timestamp = datetime.now()
# GLM智能：基础Future实现
class BasicFutureManager:
"""基础Future管理器"""
def __init__(self, max_workers: int = 4):
self.executor = ThreadPoolExecutor(max_workers=max_workers)
self.active_futures = {}
self.completed_tasks = []
def submit_task(self, task_id: str, func: Callable, *args, **kwargs) -> Any:
"""提交任务并返回Future"""
if task_id in self.active_futures:
raise ValueError(f"任务ID {task_id} 已存在")
future = self.executor.submit(func, *args, **kwargs)
self.active_futures[task_id] = future
# 添加完成回调
future.add_done_callback(lambda f: self._task_completed(task_id, f))
return future
def get_task_result(self, task_id: str, timeout: Optional[float] = None) -> Any:
"""获取任务结果"""
if task_id not in self.active_futures:
raise ValueError(f"任务ID {task_id} 不存在")
future = self.active_futures[task_id]
try:
result = future.result(timeout=timeout)
return result
except Exception as e:
raise RuntimeError(f"获取任务 {task_id} 结果失败: {e}")
def cancel_task(self, task_id: str) -> bool:
"""取消任务"""
if task_id not in self.active_futures:
return False
future = self.active_futures[task_id]
cancelled = future.cancel()
if cancelled:
del self.active_futures[task_id]
return cancelled
def wait_for_tasks(self, task_ids: List[str], timeout: Optional[float] = None) -> Dict[str, bool]:
"""等待多个任务完成"""
futures = {}
for task_id in task_ids:
if task_id in self.active_futures:
futures[task_id] = self.active_futures[task_id]
if not futures:
return {}
completed_futures = wait(futures.values(), timeout=timeout)
result = {}
for task_id, future in futures.items():
result[task_id] = future in completed_futures.done
return result
def get_task_status(self, task_id: str) -> TaskStatus:
"""获取任务状态"""
if task_id not in self.active_futures:
return TaskStatus.COMPLETED # 假设已完成并被清理
future = self.active_futures[task_id]
if future.cancelled():
return TaskStatus.CANCELLED
elif future.running():
return TaskStatus.RUNNING
elif future.done():
return TaskStatus.COMPLETED
else:
return TaskStatus.PENDING
def cleanup_completed(self):
"""清理已完成的任务"""
completed_ids = []
for task_id, future in self.active_futures.items():
if future.done():
completed_ids.append(task_id)
for task_id in completed_ids:
del self.active_futures[task_id]
def shutdown(self, wait: bool = True):
"""关闭执行器"""
self.executor.shutdown(wait=wait)
def _task_completed(self, task_id: str, future: Any):
"""任务完成回调"""
try:
result = future.result()
task_result = TaskResult(
task_id=task_id,
task_type=TaskType.COMPUTATION,
result=result,
execution_time=0.0, # 需要实际计算
success=True
)
self.completed_tasks.append(task_result)
except Exception as e:
task_result = TaskResult(
task_id=task_id,
task_type=TaskType.COMPUTATION,
result=None,
execution_time=0.0,
success=False,
error=str(e)
)
self.completed_tasks.append(task_result)
# GLM智能：高级Future管理器
class AdvancedFutureManager:
"""高级Future管理器，支持链式调用和组合"""
def __init__(self, max_workers: int = 4):
self.executor = ThreadPoolExecutor(max_workers=max_workers)
self.pipeline_futures = {}
self.task_dependencies = {}
def submit_task(self, task_id: str, func: Callable, *args, **kwargs) -> Any:
"""提交基础任务"""
future = self.executor.submit(func, *args, **kwargs)
return future
def submit_chained_task(self, task_id: str, tasks: List[Callable]) -> Any:
"""提交链式任务"""
if not tasks:
raise ValueError("任务链不能为空")
def execute_chain():
result = None
for i, task in enumerate(tasks):
if callable(task):
if i == 0:
result = task()
else:
result = task(result)
else:
raise ValueError(f"任务链中的第{i+1}个元素不是可调用对象")
return result
future = self.executor.submit(execute_chain)
self.pipeline_futures[task_id] = future
return future
def submit_parallel_tasks(self, group_id: str, tasks: Dict[str, Callable]) -> Dict[str, Any]:
"""提交并行任务组"""
futures = {}
for task_id, func in tasks.items():
future = self.executor.submit(func)
futures[task_id] = future
# 存储任务组关系
self.task_dependencies[group_id] = list(futures.keys())
return futures
def submit_dependent_task(self, task_id: str, dependency_ids: List[str],
func: Callable, *args, **kwargs) -> Any:
"""提交依赖任务"""
if dependency_ids:
# 创建等待依赖完成的函数
def wait_and_execute():
# 这里应该等待依赖任务完成，简化实现
time.sleep(0.1) # 模拟等待
return func(*args, **kwargs)
future = self.executor.submit(wait_and_execute)
else:
future = self.executor.submit(func, *args, **kwargs)
return future
def create_future_chain(self, initial_future: Any, *callbacks: Callable) -> Any:
"""创建Future链"""
current_future = initial_future
for callback in callbacks:
def create_next_callback(current_fut, cb):
def next_callback(fut):
try:
result = fut.result()
return cb(result)
except Exception as e:
raise e
return next_callback
# 由于concurrent.futures.Future不支持链式调用，这里简化实现
# 实际应用中可能需要使用asyncio.Future或自定义Future实现
pass
return current_future
def aggregate_results(self, futures: Dict[str, Any],
aggregator: Callable = None) -> Dict[str, Any]:
"""聚合多个Future的结果"""
results = {}
for task_id, future in futures.items():
try:
result = future.result()
results[task_id] = result
except Exception as e:
results[task_id] = {'error': str(e)}
if aggregator:
return aggregator(results)
return results
# GLM智能：网络请求Future管理器
class NetworkFutureManager:
"""专门处理网络请求的Future管理器"""
def __init__(self, max_workers: int = 10, timeout: float = 30.0):
self.executor = ThreadPoolExecutor(max_workers=max_workers)
self.timeout = timeout
self.session = requests.Session()
def fetch_url(self, url: str, method: str = 'GET', **kwargs) -> Dict[str, Any]:
"""获取URL内容"""
def fetch():
try:
response = self.session.request(method, url, timeout=self.timeout, **kwargs)
response.raise_for_status()
return {
'url': url,
'status_code': response.status_code,
'content': response.text[:1000], # 限制内容长度
'headers': dict(response.headers),
'success': True
}
except Exception as e:
return {
'url': url,
'error': str(e),
'success': False
}
future = self.executor.submit(fetch)
return future
def fetch_multiple_urls(self, urls: List[str], method: str = 'GET',
**kwargs) -> Dict[str, Any]:
"""批量获取多个URL"""
futures = {}
for url in urls:
future = self.fetch_url(url, method, **kwargs)
futures[url] = future
return futures
def fetch_with_retry(self, url: str, max_retries: int = 3, delay: float = 1.0) -> Any:
"""带重试的URL获取"""
def fetch_with_retry_func():
for attempt in range(max_retries):
try:
response = self.session.get(url, timeout=self.timeout)
response.raise_for_status()
return {
'url': url,
'status_code': response.status_code,
'content': response.text[:1000],
'attempt': attempt + 1,
'success': True
}
except Exception as e:
if attempt == max_retries - 1:
return {
'url': url,
'error': str(e),
'attempts': max_retries,
'success': False
}
time.sleep(delay * (2 ** attempt)) # 指数退避
future = self.executor.submit(fetch_with_retry_func)
return future
def close(self):
"""关闭管理器"""
self.session.close()
self.executor.shutdown(wait=True)
# GLM智能：计算密集型Future管理器
class ComputeFutureManager:
"""计算密集型任务Future管理器"""
def __init__(self, use_processes: bool = True, max_workers: int = None):
if use_processes:
self.executor = ProcessPoolExecutor(max_workers=max_workers)
else:
self.executor = ThreadPoolExecutor(max_workers=max_workers)
self.use_processes = use_processes
def fibonacci(self, n: int) -> int:
"""计算斐波那契数列"""
if n <= 1:
return n
return self.fibonacci(n-1) + self.fibonacci(n-2)
def submit_fibonacci(self, n: int, task_id: str = None) -> Any:
"""提交斐波那契计算任务"""
def compute():
start_time = time.time()
result = self.fibonacci(n)
execution_time = time.time() - start_time
return {
'input': n,
'result': result,
'execution_time': execution_time,
'task_id': task_id or f"fib_{n}"
}
future = self.executor.submit(compute)
return future
def matrix_multiply(self, matrix_a: List[List[float]],
matrix_b: List[List[float]]) -> List[List[float]]:
"""矩阵乘法"""
rows_a, cols_a = len(matrix_a), len(matrix_a[0])
rows_b, cols_b = len(matrix_b), len(matrix_b[0])
if cols_a != rows_b:
raise ValueError("矩阵维度不匹配")
result = [[0.0 for _ in range(cols_b)] for _ in range(rows_a)]
for i in range(rows_a):
for j in range(cols_b):
sum_val = 0.0
for k in range(cols_a):
sum_val += matrix_a[i][k] * matrix_b[k][j]
result[i][j] = sum_val
return result
def submit_matrix_multiply(self, matrix_a: List[List[float]],
matrix_b: List[List[float]], task_id: str = None) -> Any:
"""提交矩阵乘法任务"""
def compute():
start_time = time.time()
result = self.matrix_multiply(matrix_a, matrix_b)
execution_time = time.time() - start_time
return {
'matrix_a_dims': (len(matrix_a), len(matrix_a[0])),
'matrix_b_dims': (len(matrix_b), len(matrix_b[0])),
'result_dims': (len(result), len(result[0])),
'result': result,
'execution_time': execution_time,
'task_id': task_id or "matrix_multiply"
}
future = self.executor.submit(compute)
return future
def shutdown(self):
"""关闭执行器"""
self.executor.shutdown(wait=True)
# GLM智能：任务示例函数
def cpu_intensive_task(n: int) -> Dict[str, Any]:
"""CPU密集型任务示例"""
start_time = time.time()
# 模拟复杂计算
result = 0
for i in range(n):
result += i ** 2
if i % 1000000 == 0:
time.sleep(0.001) # 防止过度占用CPU
execution_time = time.time() - start_time
return {
'input': n,
'result': result,
'execution_time': execution_time
}
def io_simulation_task(file_size: int) -> Dict[str, Any]:
"""IO模拟任务"""
start_time = time.time()
# 模拟文件读取
time.sleep(file_size / 1000.0) # 模拟读取时间
# 模拟数据处理
data = list(range(file_size // 100))
processed_data = [x * 2 for x in data]
execution_time = time.time() - start_time
return {
'file_size': file_size,
'data_points': len(processed_data),
'execution_time': execution_time
}
def network_simulation_task(url: str, delay_range: tuple = (0.1, 2.0)) -> Dict[str, Any]:
"""网络请求模拟任务"""
start_time = time.time()
# 模拟网络延迟
delay = random.uniform(*delay_range)
time.sleep(delay)
# 模拟网络响应
response_code = random.choice([200, 200, 200, 404, 500])
execution_time = time.time() - start_time
return {
'url': url,
'response_code': response_code,
'response_time': execution_time,
'success': response_code == 200
}
# GLM智能：基础Future演示
def basic_future_demo():
"""基础Future演示"""
print("=== 基础Future模式演示 ===")
manager = BasicFutureManager(max_workers=3)
# 提交不同类型的任务
tasks = []
# CPU密集型任务
cpu_future = manager.submit_task(
"cpu_task_1",
cpu_intensive_task,
1000000
)
tasks.append(("CPU密集型任务", cpu_future))
# IO模拟任务
io_future = manager.submit_task(
"io_task_1",
io_simulation_task,
5000
)
tasks.append(("IO模拟任务", io_future))
# 网络模拟任务
net_future = manager.submit_task(
"net_task_1",
network_simulation_task,
"https://example.com/api/data"
)
tasks.append(("网络模拟任务", net_future))
# 检查任务状态
print("\n任务状态检查:")
for task_name, future in tasks:
print(f"{task_name}: running={future.running()}, done={future.done()}")
# 等待任务完成
print("\n等待任务完成...")
for task_name, future in tasks:
try:
result = future.result(timeout=10)
print(f"{task_name} 完成: {result}")
except Exception as e:
print(f"{task_name} 失败: {e}")
# 显示完成状态
print("\n最终任务状态:")
for task_name, future in tasks:
print(f"{task_name}: done={future.done()}, cancelled={future.cancelled()}")
manager.shutdown()
# GLM智能：批量Future演示
def batch_future_demo():
"""批量Future演示"""
print("\n=== 批量Future模式演示 ===")
manager = BasicFutureManager(max_workers=5)
# 提交批量任务
batch_futures = {}
for i in range(8):
task_id = f"batch_task_{i}"
future = manager.submit_task(
task_id,
cpu_intensive_task,
random.randint(100000, 500000)
)
batch_futures[task_id] = future
print(f"提交了 {len(batch_futures)} 个批量任务")
# 使用as_completed处理完成的任务
completed_count = 0
for future in as_completed(batch_futures.values(), timeout=30):
try:
result = future.result()
completed_count += 1
print(f"批量任务 {completed_count} 完成: result={result['execution_time']:.3f}s")
except Exception as e:
print(f"批量任务失败: {e}")
print(f"总共完成了 {completed_count} 个任务")
manager.shutdown()
# GLM智能：网络Future演示
def network_future_demo():
"""网络Future演示"""
print("\n=== 网络Future模式演示 ===")
net_manager = NetworkFutureManager(max_workers=3, timeout=5.0)
# 测试URL列表
test_urls = [
"https://httpbin.org/get",
"https://httpbin.org/delay/1",
"https://httpbin.org/status/200",
"https://httpbin.org/json"
]
# 批量获取URL
print("批量获取URL内容...")
futures = net_manager.fetch_multiple_urls(test_urls)
# 处理结果
for url, future in futures.items():
try:
result = future.result(timeout=10)
status = "成功" if result['success'] else "失败"
print(f"{url}: {status} (状态码: {result.get('status_code', 'N/A')})")
except Exception as e:
print(f"{url}: 获取失败 - {e}")
# 测试带重试的请求
print("\n测试带重试的请求...")
retry_future = net_manager.fetch_with_retry("https://httpbin.org/delay/2", max_retries=2)
try:
retry_result = retry_future.result(timeout=10)
print(f"重试请求结果: {retry_result}")
except Exception as e:
print(f"重试请求失败: {e}")
net_manager.close()
# GLM智能：计算Future演示
def compute_future_demo():
"""计算Future演示"""
print("\n=== 计算Future模式演示 ===")
compute_manager = ComputeFutureManager(use_processes=True, max_workers=2)
# 斐波那契计算
print("提交斐波那契计算任务...")
fib_futures = []
for n in [20, 25, 30, 35]:
future = compute_manager.submit_fibonacci(n, f"fib_{n}")
fib_futures.append((n, future))
for n, future in fib_futures:
try:
result = future.result(timeout=30)
print(f"fibonacci({n}) = {result['result']} (耗时: {result['execution_time']:.3f}s)")
except Exception as e:
print(f"fibonacci({n}) 计算失败: {e}")
# 矩阵乘法计算
print("\n提交矩阵乘法任务...")
matrix_a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
matrix_b = [[9, 8, 7], [6, 5, 4], [3, 2, 1]]
matrix_future = compute_manager.submit_matrix_multiply(matrix_a, matrix_b)
try:
matrix_result = matrix_future.result(timeout=10)
print(f"矩阵乘法完成 (耗时: {matrix_result['execution_time']:.3f}s)")
print(f"结果矩阵大小: {matrix_result['result_dims']}")
except Exception as e:
print(f"矩阵乘法失败: {e}")
compute_manager.shutdown()
# GLM智能：Future链式演示
def future_chain_demo():
"""Future链式演示"""
print("\n=== Future链式模式演示 ===")
manager = AdvancedFutureManager(max_workers=2)
# 创建数据处理链
def generate_data():
return list(range(1, 101))
def filter_even(data):
return [x for x in data if x % 2 == 0]
def calculate_sum(data):
return sum(data)
def calculate_average(data):
return len(data), sum(data) / len(data) if data else 0
# 提交链式任务
print("提交数据处理链...")
chain_future = manager.submit_chained_task(
"data_processing_chain",
[generate_data, filter_even, calculate_sum]
)
try:
chain_result = chain_future.result(timeout=10)
print(f"链式任务结果: 1-100中偶数之和 = {chain_result}")
except Exception as e:
print(f"链式任务失败: {e}")
# 提交并行任务组
print("\n提交并行任务组...")
parallel_tasks = {
"sum_task": lambda: sum(range(1, 501)),
"factorial_task": lambda: functools.reduce(lambda x, y: x*y, range(1, 11)),
"prime_task": lambda: len([x for x in range(2, 100) if all(x % y != 0 for y in range(2, int(x**0.5)+1))])
}
parallel_futures = manager.submit_parallel_tasks("math_group", parallel_tasks)
# 聚合结果
def aggregate_math_results(results):
summary = {
'total_tasks': len(results),
'successful_tasks': sum(1 for r in results.values() if isinstance(r, (int, float))),
'results': results
}
return summary
try:
# 等待所有并行任务完成
aggregated = manager.aggregate_results(parallel_futures, aggregate_math_results)
print(f"并行任务聚合结果:")
print(f" 总任务数: {aggregated['total_tasks']}")
print(f" 成功任务数: {aggregated['successful_tasks']}")
print(f" 结果: {aggregated['results']}")
except Exception as e:
print(f"并行任务聚合失败: {e}")
manager.executor.shutdown()
# GLM智能：Future超时和取消演示
def future_timeout_cancel_demo():
"""Future超时和取消演示"""
print("\n=== Future超时和取消演示 ===")
manager = BasicFutureManager(max_workers=2)
# 提交长时间运行的任务
long_task_future = manager.submit_task(
"long_task",
lambda: time.sleep(10) or "完成"
)
print("提交长时间运行任务...")
# 尝试带超时获取结果
print("尝试3秒超时获取结果...")
try:
result = manager.get_task_result("long_task", timeout=3)
print(f"任务结果: {result}")
except Exception as e:
print(f"超时获取失败: {e}")
# 取消任务
print("尝试取消任务...")
cancelled = manager.cancel_task("long_task")
print(f"任务取消结果: {cancelled}")
# 检查任务状态
status = manager.get_task_status("long_task")
print(f"任务最终状态: {status.value}")
# 测试快速任务
print("\n测试快速任务...")
quick_future = manager.submit_task("quick_task", lambda: "快速完成")
try:
result = manager.get_task_result("quick_task", timeout=5)
print(f"快速任务结果: {result}")
except Exception as e:
print(f"快速任务失败: {e}")
manager.shutdown()
# GLM智能：主演示函数
def main():
"""主演示函数"""
print("=== Future模式完整演示 ===\n")
# 1. 基础Future演示
basic_future_demo()
# 2. 批量Future演示
batch_future_demo()
# 3. 网络Future演示
network_future_demo()
# 4. 计算Future演示
compute_future_demo()
# 5. Future链式演示
future_chain_demo()
# 6. Future超时和取消演示
future_timeout_cancel_demo()
print("\n=== Future模式演示完成 ===")
# 运行演示
if __name__ == "__main__":
main()
---
8.4 Pipeline模式
01.Pipeline模式概述
a.基本概念
Pipeline模式是一种并发设计模式，将复杂任务分解为多个处理阶段。
每个阶段由独立的处理单元负责，通过队列连接各个阶段。
数据像流水线一样在各个阶段间流动，实现高效的并行处理。
适用于可以分解为顺序步骤的批量数据处理任务。
可以显著提升CPU利用率和系统吞吐量。
b.核心特性
a.特性说明
支持模块化设计，每个处理阶段职责单一，便于维护和扩展。
各阶段并行执行，最大化利用多核CPU资源。
通过队列缓冲，平滑处理各阶段间的速度差异。
提供背压控制机制，防止内存溢出。
支持流水线停机、恢复和监控功能。
b.优势分析
相比批量处理，Pipeline模式具有更低的延迟和更好的资源利用率。
各阶段可以独立扩展，根据瓶颈调整并发度。
错误隔离能力强，单个阶段失败不影响其他阶段。
支持动态负载均衡，根据系统负载自动调整处理速度。
c.应用场景
数据ETL处理、图像处理流水线、日志分析、实时数据处理等场景。
电商订单处理、音视频转码、机器学习特征工程等领域。
02.Pipeline模式实现
a.基础Pipeline实现
a.功能说明
实现一个可复用的Pipeline框架，支持任意数量的处理阶段。
每个阶段可以设置不同的并发度，支持自定义处理函数。
提供完整的错误处理和性能监控机制。
支持优雅关闭和资源清理，确保数据完整性。
b.代码示例
---
# Pipeline模式完整实现
import asyncio
import time
import logging
from typing import Any, Callable, List, Optional, Dict
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from queue import Queue, Empty
from threading import Thread, Event
import signal
import sys
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
@dataclass
class PipelineMetrics:
"""Pipeline性能指标"""
total_processed: int = 0
total_failed: int = 0
avg_processing_time: float = 0.0
stage_queue_sizes: Dict[str, int] = None
stage_processing_times: Dict[str, List[float]] = None
def __post_init__(self):
if self.stage_queue_sizes is None:
self.stage_queue_sizes = {}
if self.stage_processing_times is None:
self.stage_processing_times = {}
class PipelineStage:
"""Pipeline处理阶段"""
def __init__(
self,
name: str,
process_func: Callable[[Any], Any],
max_workers: int = 1,
queue_size: int = 1000,
retry_count: int = 3
):
self.name = name
self.process_func = process_func
self.max_workers = max_workers
self.queue_size = queue_size
self.retry_count = retry_count
# 队列和线程池
self.input_queue = asyncio.Queue(maxsize=queue_size)
self.output_queue = asyncio.Queue(maxsize=queue_size)
self.executor = ThreadPoolExecutor(max_workers=max_workers)
# 状态控制
self.is_running = False
self.stop_event = Event()
self.workers = []
# 性能统计
self.processed_count = 0
self.failed_count = 0
self.processing_times = []
logger.info(f"Pipeline阶段 '{name}' 初始化完成 (workers={max_workers}, queue_size={queue_size})")
async def start(self):
"""启动处理阶段"""
if self.is_running:
logger.warning(f"Pipeline阶段 '{self.name}' 已在运行")
return
self.is_running = True
self.stop_event.clear()
# 创建并启动工作协程
for i in range(self.max_workers):
worker = asyncio.create_task(self._worker(f"{self.name}-worker-{i}"))
self.workers.append(worker)
logger.info(f"Pipeline阶段 '{self.name}' 已启动，创建了 {len(self.workers)} 个工作协程")
async def stop(self):
"""停止处理阶段"""
if not self.is_running:
return
logger.info(f"正在停止Pipeline阶段 '{self.name}'...")
# 发送停止信号
self.is_running = False
self.stop_event.set()
# 等待所有工作协程完成
if self.workers:
await asyncio.gather(*self.workers, return_exceptions=True)
self.workers.clear()
# 关闭线程池
self.executor.shutdown(wait=True)
logger.info(f"Pipeline阶段 '{self.name}' 已停止")
async def _worker(self, worker_name: str):
"""工作协程"""
logger.info(f"工作协程 {worker_name} 已启动")
while self.is_running:
try:
# 从输入队列获取数据（带超时）
data = await asyncio.wait_for(
self.input_queue.get(),
timeout=1.0
)
start_time = time.time()
try:
# 处理数据（在线程池中执行）
result = await asyncio.get_event_loop().run_in_executor(
self.executor,
self._process_with_retry,
data
)
# 将结果放入输出队列
await self.output_queue.put(result)
# 更新统计
self.processed_count += 1
processing_time = time.time() - start_time
self.processing_times.append(processing_time)
if self.processed_count % 100 == 0:
avg_time = sum(self.processing_times[-100:]) / min(len(self.processing_times), 100)
logger.info(f"{worker_name}: 已处理 {self.processed_count} 项，平均耗时 {avg_time:.3f}s")
except Exception as e:
self.failed_count += 1
logger.error(f"{worker_name} 处理失败: {e}")
# 根据配置决定是否继续处理
if self.failed_count > self.processed_count * 0.1: # 失败率超过10%
logger.error(f"{worker_name}: 失败率过高，停止处理")
break
finally:
self.input_queue.task_done()
except asyncio.TimeoutError:
# 超时继续，检查是否需要停止
continue
except Exception as e:
logger.error(f"{worker_name} 发生未预期错误: {e}")
break
logger.info(f"工作协程 {worker_name} 已停止")
def _process_with_retry(self, data: Any) -> Any:
"""带重试的数据处理"""
last_exception = None
for attempt in range(self.retry_count + 1):
try:
return self.process_func(data)
except Exception as e:
last_exception = e
if attempt < self.retry_count:
logger.warning(f"处理失败 (尝试 {attempt + 1}/{self.retry_count + 1}): {e}")
time.sleep(0.1 * (2 ** attempt)) # 指数退避
else:
logger.error(f"处理最终失败: {e}")
raise
raise last_exception
async def put(self, data: Any):
"""向输入队列放入数据"""
await self.input_queue.put(data)
async def get(self) -> Any:
"""从输出队列获取数据"""
return await self.output_queue.get()
def get_queue_size(self) -> Dict[str, int]:
"""获取队列大小"""
return {
"input": self.input_queue.qsize(),
"output": self.output_queue.qsize()
}
def get_stats(self) -> Dict[str, Any]:
"""获取处理统计"""
return {
"processed": self.processed_count,
"failed": self.failed_count,
"success_rate": self.processed_count / max(1, self.processed_count + self.failed_count),
"avg_processing_time": sum(self.processing_times) / max(1, len(self.processing_times)) if self.processing_times else 0,
"queue_sizes": self.get_queue_size()
}
class Pipeline:
"""Pipeline管理器"""
def __init__(self, name: str):
self.name = name
self.stages: List[PipelineStage] = []
self.is_running = False
# 性能监控
self.metrics = PipelineMetrics()
self.start_time: Optional[float] = None
self.end_time: Optional[float] = None
logger.info(f"Pipeline '{name}' 已创建")
def add_stage(self, stage: PipelineStage) -> 'Pipeline':
"""添加处理阶段"""
self.stages.append(stage)
logger.info(f"已向Pipeline '{self.name}' 添加阶段 '{stage.name}'")
return self
def add_stage_config(
self,
name: str,
process_func: Callable[[Any], Any],
max_workers: int = 1,
queue_size: int = 1000,
retry_count: int = 3
) -> 'Pipeline':
"""便捷方法：添加配置好的阶段"""
stage = PipelineStage(name, process_func, max_workers, queue_size, retry_count)
return self.add_stage(stage)
async def start(self):
"""启动Pipeline"""
if self.is_running:
logger.warning(f"Pipeline '{self.name}' 已在运行")
return
if not self.stages:
raise ValueError(f"Pipeline '{self.name}' 没有配置任何阶段")
self.start_time = time.time()
self.is_running = True
# 启动所有阶段
for stage in self.stages:
await stage.start()
# 连接相邻阶段的队列
for i in range(len(self.stages) - 1):
current_stage = self.stages[i]
next_stage = self.stages[i + 1]
# 创建数据传输协程
transfer_task = asyncio.create_task(
self._transfer_data(current_stage, next_stage)
)
logger.info(f"Pipeline '{self.name}' 已启动，包含 {len(self.stages)} 个阶段")
async def stop(self):
"""停止Pipeline"""
if not self.is_running:
return
logger.info(f"正在停止Pipeline '{self.name}'...")
self.is_running = False
self.end_time = time.time()
# 按相反顺序停止所有阶段
for stage in reversed(self.stages):
await stage.stop()
# 计算最终指标
if self.start_time and self.end_time:
total_time = self.end_time - self.start_time
self.metrics.avg_processing_time = total_time / max(1, self.metrics.total_processed)
logger.info(f"Pipeline '{self.name}' 已停止")
async def _transfer_data(self, from_stage: PipelineStage, to_stage: PipelineStage):
"""在两个阶段间传输数据"""
logger.info(f"开始从 '{from_stage.name}' 向 '{to_stage.name}' 传输数据")
while from_stage.is_running or not from_stage.input_queue.empty():
try:
# 从前一个阶段获取数据
data = await asyncio.wait_for(
from_stage.output_queue.get(),
timeout=1.0
)
# 传递给下一个阶段
await to_stage.put(data)
from_stage.output_queue.task_done()
except asyncio.TimeoutError:
# 检查前一个阶段是否还在运行
if not from_stage.is_running and from_stage.output_queue.empty():
break
continue
except Exception as e:
logger.error(f"数据传输失败 ({from_stage.name} -> {to_stage.name}): {e}")
break
logger.info(f"数据传输完成 ({from_stage.name} -> {to_stage.name})")
async def process(self, data: Any):
"""处理单个数据项"""
if not self.is_running:
raise RuntimeError(f"Pipeline '{self.name}' 未运行")
# 将数据放入第一个阶段
await self.stages[0].put(data)
# 从最后一个阶段获取结果
return await self.stages[-1].get()
async def process_batch(self, data_list: List[Any]):
"""批量处理数据"""
if not self.is_running:
raise RuntimeError(f"Pipeline '{self.name}' 未运行")
results = []
# 将所有数据放入第一个阶段
for data in data_list:
await self.stages[0].put(data)
# 从最后一个阶段获取所有结果
for _ in range(len(data_list)):
result = await self.stages[-1].get()
results.append(result)
return results
def get_metrics(self) -> PipelineMetrics:
"""获取Pipeline性能指标"""
# 更新总体指标
self.metrics.total_processed = sum(stage.processed_count for stage in self.stages)
self.metrics.total_failed = sum(stage.failed_count for stage in self.stages)
# 更新各阶段指标
for stage in self.stages:
self.metrics.stage_queue_sizes[stage.name] = stage.input_queue.qsize()
self.metrics.stage_processing_times[stage.name] = stage.processing_times.copy()
return self.metrics
def get_detailed_stats(self) -> Dict[str, Any]:
"""获取详细统计信息"""
stats = {
"pipeline_name": self.name,
"is_running": self.is_running,
"stages": len(self.stages),
"total_metrics": self.get_metrics().__dict__
}
if self.start_time:
stats["uptime"] = time.time() - self.start_time
# 添加各阶段详细信息
stats["stage_details"] = {}
for stage in self.stages:
stats["stage_details"][stage.name] = stage.get_stats()
return stats
# 示例使用：数据处理Pipeline
async def main():
"""Pipeline使用示例"""
# 定义处理函数
def data_loader(data):
"""数据加载阶段"""
time.sleep(0.1) # 模拟IO操作
return {"raw": data, "loaded_at": time.time()}
def data_validator(item):
"""数据验证阶段"""
time.sleep(0.05) # 模拟验证逻辑
if not item.get("raw"):
raise ValueError("数据为空")
return {**item, "validated": True, "validated_at": time.time()}
def data_transformer(item):
"""数据转换阶段"""
time.sleep(0.2) # 模拟复杂转换
raw_data = item["raw"]
transformed = f"processed_{raw_data}_{time.time()}"
return {**item, "transformed": transformed, "transformed_at": time.time()}
def data_saver(item):
"""数据保存阶段"""
time.sleep(0.15) # 模拟保存操作
return {**item, "saved": True, "saved_at": time.time()}
# 创建Pipeline
pipeline = Pipeline("data-processing-pipeline")
# 添加处理阶段
pipeline.add_stage_config("loader", data_loader, max_workers=2)
pipeline.add_stage_config("validator", data_validator, max_workers=3)
pipeline.add_stage_config("transformer", data_transformer, max_workers=2)
pipeline.add_stage_config("saver", data_saver, max_workers=1)
# 设置优雅关闭
def signal_handler(sig, frame):
logger.info("接收到关闭信号，正在优雅关闭...")
asyncio.create_task(pipeline.stop())
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
try:
# 启动Pipeline
await pipeline.start()
# 生成测试数据
test_data = [f"data_{i}" for i in range(100)]
logger.info(f"开始处理 {len(test_data)} 项数据...")
# 批量处理数据
start_time = time.time()
results = await pipeline.process_batch(test_data)
end_time = time.time()
# 输出结果统计
logger.info(f"处理完成！")
logger.info(f"总耗时: {end_time - start_time:.2f} 秒")
logger.info(f"处理速度: {len(test_data) / (end_time - start_time):.2f} 项/秒")
logger.info(f"成功处理: {len(results)} 项")
# 输出详细统计
stats = pipeline.get_detailed_stats()
logger.info("Pipeline统计信息:")
for stage_name, stage_stats in stats["stage_details"].items():
logger.info(f" {stage_name}: {stage_stats}")
# 输出性能指标
metrics = pipeline.get_metrics()
logger.info(f"总体指标: 总处理={metrics.total_processed}, 总失败={metrics.total_failed}")
except Exception as e:
logger.error(f"Pipeline执行失败: {e}")
finally:
# 停止Pipeline
await pipeline.stop()
if __name__ == "__main__":
# 运行Pipeline示例
asyncio.run(main())
---
03.Pipeline模式优化
a.性能优化策略
a.队列大小调优
根据内存容量和处理速度差异，合理设置各阶段队列大小。
使用背压机制防止内存溢出，实现流式处理。
动态调整队列大小，根据系统负载自适应优化。
b.并发度优化
分析各阶段的CPU密集型和IO密集型特性，设置合适的并发度。
使用性能监控工具识别瓶颈阶段，针对性地增加工作协程。
实现动态负载均衡，根据队列长度自动调整工作协程数量。
b.监控与调试
a.实时监控
实现Pipeline性能仪表板，实时监控各阶段处理状态。
收集关键指标：吞吐量、延迟、错误率、资源利用率。
设置告警机制，及时发现异常情况。
b.调试工具
提供Pipeline可视化工具，直观展示数据流动和处理状态。
实现断点调试功能，支持单步跟踪数据处理过程。
提供性能分析报告，帮助识别优化机会。
8.5 Actor模型
01.Actor模型概述
a.基本概念
Actor模型是一种并发计算模型，由Carl Hewitt在1973年提出。
系统由多个独立的Actor组成，每个Actor是并发计算的基本单元。
Actor之间通过异步消息传递进行通信，不共享内存状态。
每个Actor有自己的邮箱，按顺序处理接收到的消息。
提供天然的并发安全性和容错能力。
b.核心特性
a.状态封装
每个Actor封装自己的状态，外部无法直接访问。
状态只能通过发送消息来修改，避免并发冲突。
提供强大的数据隔离和保护机制。
b.消息传递
Actor之间通过不可变的消息进行通信。
消息传递是异步的，发送者不需要等待接收者响应。
支持请求-响应模式和发布-订阅模式。
c.行为封装
每个Actor定义自己的行为逻辑，如何处理不同类型的消息。
支持状态转换，可以根据接收的消息改变行为。
提供灵活的协议定义和错误处理机制。
c.优势分析
避免了传统并发编程中的死锁、竞态条件等问题。
天然支持分布式系统，Actor可以跨网络部署。
提供优秀的可扩展性和容错性，支持系统动态增减节点。
简化了并发编程的复杂性，提高了代码的可维护性。
02.Actor模型实现
a.基础Actor实现
a.功能说明
实现一个完整的Actor框架，支持消息发送、接收和处理。
提供Actor生命周期管理，包括创建、启动、暂停、恢复和销毁。
实现Actor监督机制，支持父子Actor关系和错误恢复。
提供Actor注册发现功能，支持Actor目录和路由。
b.代码示例
---
# Actor模型完整实现
import asyncio
import time
import logging
import uuid
from typing import Any, Callable, Dict, List, Optional, Union, Type
from dataclasses import dataclass, field
from enum import Enum
from abc import ABC, abstractmethod
import traceback
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class ActorState(Enum):
"""Actor状态枚举"""
CREATED = "created"
STARTING = "starting"
RUNNING = "running"
PAUSED = "paused"
STOPPING = "stopping"
STOPPED = "stopped"
CRASHED = "crashed"
@dataclass
class ActorMessage:
"""Actor消息"""
sender_id: str
message_type: str
content: Any
timestamp: float = field(default_factory=time.time)
correlation_id: Optional[str] = None
timeout: Optional[float] = None
@dataclass
class ActorRef:
"""Actor引用"""
actor_id: str
actor_system: 'ActorSystem'
def send_message(self, message_type: str, content: Any, timeout: Optional[float] = None):
"""发送消息给Actor"""
message = ActorMessage(
sender_id="system",
message_type=message_type,
content=content,
timeout=timeout
)
return self.actor_system.send_message(self.actor_id, message)
def send_message_with_reply(self, message_type: str, content: Any, timeout: float = 5.0) -> Any:
"""发送消息并等待回复"""
correlation_id = str(uuid.uuid4())
message = ActorMessage(
sender_id=self.actor_id,
message_type=message_type,
content=content,
correlation_id=correlation_id,
timeout=timeout
)
return self.actor_system.send_message_with_reply(self.actor_id, message, timeout)
async def send_message_async(self, message_type: str, content: Any, timeout: Optional[float] = None):
"""异步发送消息"""
message = ActorMessage(
sender_id="system",
message_type=message_type,
content=content,
timeout=timeout
)
await self.actor_system.send_message_async(self.actor_id, message)
class Actor(ABC):
"""抽象Actor基类"""
def __init__(self, actor_id: str, actor_system: 'ActorSystem'):
self.actor_id = actor_id
self.actor_system = actor_system
self.state = ActorState.CREATED
self.mailbox = asyncio.Queue(maxsize=1000)
self.supervisor: Optional['Actor'] = None
self.children: List['Actor'] = []
self.context: Dict[str, Any] = {}
# 性能统计
self.messages_processed = 0
self.messages_failed = 0
self.start_time: Optional[float] = None
self.last_message_time: Optional[float] = None
# 事件循环和任务
self.loop = asyncio.get_event_loop()
self.processing_task: Optional[asyncio.Task] = None
self.supervision_task: Optional[asyncio.Task] = None
logger.info(f"Actor {actor_id} 已创建")
@abstractmethod
async def receive(self, message: ActorMessage) -> Any:
"""处理接收到的消息（子类必须实现）"""
pass
async def start(self):
"""启动Actor"""
if self.state != ActorState.CREATED:
logger.warning(f"Actor {self.actor_id} 状态为 {self.state.value}，无法启动")
return
self.state = ActorState.STARTING
self.start_time = time.time()
try:
await self.on_start()
# 启动消息处理任务
self.processing_task = asyncio.create_task(self._process_messages())
# 启动子Actor监督任务
if self.children:
self.supervision_task = asyncio.create_task(self._supervise_children())
self.state = ActorState.RUNNING
logger.info(f"Actor {self.actor_id} 已启动")
except Exception as e:
self.state = ActorState.CRASHED
logger.error(f"Actor {self.actor_id} 启动失败: {e}")
await self._notify_supervisor(f"actor_crashed", {"error": str(e), "traceback": traceback.format_exc()})
async def stop(self, timeout: float = 10.0):
"""停止Actor"""
if self.state not in [ActorState.RUNNING, ActorState.PAUSED]:
return
self.state = ActorState.STOPPING
try:
# 停止所有子Actor
for child in self.children:
await child.stop(timeout)
# 等待消息处理完成
if self.processing_task:
await asyncio.wait_for(self.processing_task, timeout=timeout)
if self.supervision_task:
await asyncio.wait_for(self.supervision_task, timeout=timeout)
await self.on_stop()
self.state = ActorState.STOPPED
logger.info(f"Actor {self.actor_id} 已停止")
except asyncio.TimeoutError:
logger.warning(f"Actor {self.actor_id} 停止超时")
self.state = ActorState.CRASHED
except Exception as e:
self.state = ActorState.CRASHED
logger.error(f"Actor {self.actor_id} 停止失败: {e}")
async def pause(self):
"""暂停Actor"""
if self.state != ActorState.RUNNING:
return
self.state = ActorState.PAUSED
await self.on_pause()
logger.info(f"Actor {self.actor_id} 已暂停")
async def resume(self):
"""恢复Actor"""
if self.state != ActorState.PAUSED:
return
self.state = ActorState.RUNNING
await self.on_resume()
logger.info(f"Actor {self.actor_id} 已恢复")
def create_child(self, actor_class: Type['Actor'], actor_id: str, **kwargs) -> 'Actor':
"""创建子Actor"""
child = actor_class(actor_id, self.actor_system, **kwargs)
child.supervisor = self
self.children.append(child)
return child
async def send_message(self, message: ActorMessage):
"""发送消息到Actor邮箱"""
await self.mailbox.put(message)
async def _process_messages(self):
"""处理消息的主循环"""
logger.info(f"Actor {self.actor_id} 开始处理消息")
while self.state in [ActorState.RUNNING, ActorState.PAUSED]:
try:
if self.state == ActorState.PAUSED:
await asyncio.sleep(0.1)
continue
# 从邮箱获取消息
try:
message = await asyncio.wait_for(self.mailbox.get(), timeout=1.0)
except asyncio.TimeoutError:
continue
self.last_message_time = time.time()
try:
# 处理消息
result = await self.receive(message)
self.messages_processed += 1
# 如果有关联ID，发送回复
if message.correlation_id and message.sender_id != "system":
reply = ActorMessage(
sender_id=self.actor_id,
message_type="reply",
content=result,
correlation_id=message.correlation_id
)
await self.actor_system.send_message(message.sender_id, reply)
except Exception as e:
self.messages_failed += 1
logger.error(f"Actor {self.actor_id} 处理消息失败: {e}")
# 通知监督者
await self._notify_supervisor("message_processing_failed", {
"message": message,
"error": str(e),
"traceback": traceback.format_exc()
})
# 根据错误处理策略决定是否继续
if self.messages_failed > 10 and self.messages_failed / max(1, self.messages_processed) > 0.1:
logger.error(f"Actor {self.actor_id} 失败率过高，停止处理")
break
finally:
self.mailbox.task_done()
except Exception as e:
logger.error(f"Actor {self.actor_id} 消息处理循环异常: {e}")
break
logger.info(f"Actor {self.actor_id} 消息处理结束")
async def _supervise_children(self):
"""监督子Actor"""
while self.state in [ActorState.RUNNING, ActorState.PAUSED]:
try:
for child in self.children[:]: # 使用副本避免迭代问题
if child.state == ActorState.CRASHED:
await self.on_child_crashed(child)
# 根据监督策略处理
await self._handle_child_failure(child)
await asyncio.sleep(1.0)
except Exception as e:
logger.error(f"Actor {self.actor_id} 子Actor监督异常: {e}")
break
async def _handle_child_failure(self, child: 'Actor'):
"""处理子Actor失败"""
# 默认策略：重启子Actor
try:
logger.info(f"Actor {self.actor_id} 重启子Actor {child.actor_id}")
await child.stop()
await child.start()
except Exception as e:
logger.error(f"Actor {self.actor_id} 重启子Actor {child.actor_id} 失败: {e}")
async def _notify_supervisor(self, event_type: str, data: Dict[str, Any]):
"""通知监督者事件"""
if self.supervisor:
message = ActorMessage(
sender_id=self.actor_id,
message_type=f"supervisor_event_{event_type}",
content=data
)
await self.supervisor.send_message(message)
def get_stats(self) -> Dict[str, Any]:
"""获取Actor统计信息"""
uptime = time.time() - self.start_time if self.start_time else 0
return {
"actor_id": self.actor_id,
"state": self.state.value,
"messages_processed": self.messages_processed,
"messages_failed": self.messages_failed,
"success_rate": self.messages_processed / max(1, self.messages_processed + self.messages_failed),
"uptime": uptime,
"mailbox_size": self.mailbox.qsize(),
"children_count": len(self.children),
"last_message_time": self.last_message_time
}
# 生命周期回调方法
async def on_start(self):
"""启动时调用"""
pass
async def on_stop(self):
"""停止时调用"""
pass
async def on_pause(self):
"""暂停时调用"""
pass
async def on_resume(self):
"""恢复时调用"""
pass
async def on_child_crashed(self, child: 'Actor'):
"""子Actor崩溃时调用"""
pass
class ActorSystem:
"""Actor系统管理器"""
def __init__(self, system_name: str = "default"):
self.system_name = system_name
self.actors: Dict[str, Actor] = {}
self.actor_directory: Dict[str, ActorRef] = {}
self.reply_futures: Dict[str, asyncio.Future] = {}
# 性能统计
self.total_actors_created = 0
self.total_messages_sent = 0
logger.info(f"Actor系统 '{system_name}' 已创建")
async def start(self):
"""启动Actor系统"""
logger.info(f"Actor系统 '{self.system_name}' 已启动")
async def stop(self, timeout: float = 30.0):
"""停止Actor系统"""
logger.info(f"正在停止Actor系统 '{self.system_name}'...")
# 停止所有Actor
stop_tasks = []
for actor in self.actors.values():
stop_tasks.append(actor.stop(timeout))
await asyncio.gather(*stop_tasks, return_exceptions=True)
self.actors.clear()
self.actor_directory.clear()
self.reply_futures.clear()
logger.info(f"Actor系统 '{self.system_name}' 已停止")
def create_actor(self, actor_class: Type[Actor], actor_id: Optional[str] = None, **kwargs) -> ActorRef:
"""创建Actor"""
if not actor_id:
actor_id = f"{actor_class.__name__.lower()}_{uuid.uuid4().hex[:8]}"
if actor_id in self.actors:
raise ValueError(f"Actor {actor_id} 已存在")
actor = actor_class(actor_id, self, **kwargs)
self.actors[actor_id] = actor
self.actor_directory[actor_id] = ActorRef(actor_id, self)
self.total_actors_created += 1
logger.info(f"已创建Actor {actor_id} ({actor_class.__name__})")
return self.actor_directory[actor_id]
async def send_message(self, actor_id: str, message: ActorMessage) -> bool:
"""发送消息给指定Actor"""
try:
actor = self.actors.get(actor_id)
if not actor:
logger.warning(f"Actor {actor_id} 不存在")
return False
await actor.send_message(message)
self.total_messages_sent += 1
return True
except Exception as e:
logger.error(f"发送消息到Actor {actor_id} 失败: {e}")
return False
async def send_message_async(self, actor_id: str, message: ActorMessage):
"""异步发送消息"""
success = await self.send_message(actor_id, message)
if not success:
raise RuntimeError(f"发送消息到Actor {actor_id} 失败")
def send_message_with_reply(self, actor_id: str, message: ActorMessage, timeout: float = 5.0) -> Any:
"""发送消息并等待回复"""
# 在同步上下文中运行异步代码
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
return loop.run_until_complete(
self.send_message_with_reply_async(actor_id, message, timeout)
)
finally:
loop.close()
async def send_message_with_reply_async(self, actor_id: str, message: ActorMessage, timeout: float = 5.0) -> Any:
"""异步发送消息并等待回复"""
if not message.correlation_id:
message.correlation_id = str(uuid.uuid4())
# 创建回复Future
reply_future = asyncio.Future()
self.reply_futures[message.correlation_id] = reply_future
try:
# 发送消息
success = await self.send_message(actor_id, message)
if not success:
raise RuntimeError(f"发送消息到Actor {actor_id} 失败")
# 等待回复
reply = await asyncio.wait_for(reply_future, timeout=timeout)
return reply.content
except asyncio.TimeoutError:
raise TimeoutError(f"等待Actor {actor_id} 回复超时")
finally:
self.reply_futures.pop(message.correlation_id, None)
async def _handle_reply(self, reply: ActorMessage):
"""处理回复消息"""
if reply.correlation_id in self.reply_futures:
future = self.reply_futures[reply.correlation_id]
if not future.done():
future.set_result(reply)
def get_actor_ref(self, actor_id: str) -> Optional[ActorRef]:
"""获取Actor引用"""
return self.actor_directory.get(actor_id)
def get_all_actors(self) -> List[ActorRef]:
"""获取所有Actor引用"""
return list(self.actor_directory.values())
def get_system_stats(self) -> Dict[str, Any]:
"""获取系统统计信息"""
actor_stats = [actor.get_stats() for actor in self.actors.values()]
return {
"system_name": self.system_name,
"total_actors": len(self.actors),
"total_actors_created": self.total_actors_created,
"total_messages_sent": self.total_messages_sent,
"pending_replies": len(self.reply_futures),
"actor_states": {stat["actor_id"]: stat["state"] for stat in actor_stats}
}
# 示例Actor实现
class CounterActor(Actor):
"""计数器Actor示例"""
def __init__(self, actor_id: str, actor_system: ActorSystem):
super().__init__(actor_id, actor_system)
self.count = 0
async def receive(self, message: ActorMessage) -> Any:
"""处理消息"""
if message.message_type == "increment":
self.count += 1
return {"current_count": self.count}
elif message.message_type == "decrement":
self.count -= 1
return {"current_count": self.count}
elif message.message_type == "get_count":
return {"current_count": self.count}
elif message.message_type == "reset":
old_count = self.count
self.count = 0
return {"old_count": old_count, "current_count": self.count}
else:
return {"error": f"未知消息类型: {message.message_type}"}
class LoggerActor(Actor):
"""日志Actor示例"""
def __init__(self, actor_id: str, actor_system: ActorSystem):
super().__init__(actor_id, actor_system)
self.log_messages = []
async def receive(self, message: ActorMessage) -> Any:
"""处理消息"""
if message.message_type == "log":
log_entry = {
"timestamp": time.time(),
"sender": message.sender_id,
"content": message.content
}
self.log_messages.append(log_entry)
# 只保留最近的1000条日志
if len(self.log_messages) > 1000:
self.log_messages = self.log_messages[-1000:]
logger.info(f"日志Actor收到消息: {log_entry}")
return {"logged": True, "total_logs": len(self.log_messages)}
elif message.message_type == "get_logs":
return {"logs": self.log_messages.copy()}
elif message.message_type == "clear_logs":
old_count = len(self.log_messages)
self.log_messages.clear()
return {"cleared": True, "cleared_count": old_count}
else:
return {"error": f"未知消息类型: {message.message_type}"}
# Actor模型使用示例
async def main():
"""Actor系统使用示例"""
# 创建Actor系统
actor_system = ActorSystem("example_system")
await actor_system.start()
try:
# 创建Actor
counter_ref = actor_system.create_actor(CounterActor, "main_counter")
logger_ref = actor_system.create_actor(LoggerActor, "main_logger")
# 启动Actor
counter_actor = actor_system.actors["main_counter"]
logger_actor = actor_system.actors["main_logger"]
await counter_actor.start()
await logger_actor.start()
logger.info("开始Actor系统演示...")
# 演示计数器Actor
logger.info("=== 计数器Actor演示 ===")
# 发送增量消息
for i in range(5):
result = await counter_ref.send_message_with_reply("increment", {"step": i + 1})
logger.info(f"增量操作结果: {result}")
# 获取当前计数
result = await counter_ref.send_message_with_reply("get_count", {})
logger.info(f"当前计数: {result}")
# 演示日志Actor
logger.info("=== 日志Actor演示 ===")
# 发送日志消息
log_messages = [
"系统启动",
"用户登录",
"数据更新",
"系统运行正常"
]
for log_msg in log_messages:
result = await logger_ref.send_message_with_reply("log", {"message": log_msg})
logger.info(f"日志记录结果: {result}")
# 获取日志
result = await logger_ref.send_message_with_reply("get_logs", {})
logger.info(f"获取到日志条数: {len(result['logs'])}")
# Actor间通信演示
logger.info("=== Actor间通信演示 ===")
# 创建一个Actor间的通信链
def create_communication_chain():
async def chain_actor_test():
# 让计数器Actor发送日志到日志Actor
counter_stats = counter_actor.get_stats()
log_message = {
"message": f"计数器Actor统计: 处理了{counter_stats['messages_processed']}条消息"
}
# 通过Actor系统直接发送消息到日志Actor
message = ActorMessage(
sender_id="main_counter",
message_type="log",
content=log_message
)
await actor_system.send_message("main_logger", message)
asyncio.create_task(chain_actor_test())
create_communication_chain()
await asyncio.sleep(1.0) # 等待消息处理
# 输出系统统计
logger.info("=== 系统统计 ===")
system_stats = actor_system.get_system_stats()
logger.info(f"系统统计: {system_stats}")
# 输出各Actor统计
for actor in actor_system.actors.values():
actor_stats = actor.get_stats()
logger.info(f"Actor {actor.actor_id} 统计: {actor_stats}")
except Exception as e:
logger.error(f"Actor系统演示失败: {e}")
finally:
# 停止Actor系统
await actor_system.stop()
if __name__ == "__main__":
# 运行Actor系统示例
asyncio.run(main())
---
03.Actor模型高级特性
a.监督策略
a.监督层级设计
建立父子Actor监督关系，实现结构化的错误处理。
定义不同的监督策略：重启、停止、升级等。
支持自定义监督逻辑，满足特定业务需求。
b.错误恢复机制
实现指数退避重启，避免频繁重启导致的系统不稳定。
提供熔断机制，防止错误级联传播。
支持状态检查点，实现Actor状态的持久化和恢复。
b.分布式Actor
a.网络通信
扩展Actor系统支持跨网络通信。
实现Actor发现和路由机制，支持动态拓扑。
提供网络分区容错和消息重传机制。
b.集群管理
实现Actor集群的自动扩缩容。
支持负载均衡和故障转移。
提供集群状态监控和性能调优工具。
9 并发安全
9.1 线程安全的数据结构
01.线程安全数据结构概述
a.基本概念
线程安全数据结构是指可以在多线程环境中安全访问和修改的数据结构。
通过内置的同步机制确保数据一致性，避免竞态条件和数据损坏。
Python提供了多种线程安全的数据结构，如queue模块、threading.Lock等。
正确选择和使用线程安全数据结构是编写高质量并发程序的关键。
b.核心需求
a.原子性操作
确保读-改-写操作的原子性，防止中间状态被其他线程观察到。
使用锁机制保证复合操作的完整性，避免数据不一致。
提供无锁数据结构，通过原子操作实现高性能并发访问。
b.可见性保证
确保一个线程的修改对其他线程立即可见。
使用适当的内存屏障防止指令重排序导致的问题。
实现跨线程的数据一致性，保证所有线程看到相同的数据视图。
c.有序性保障
维护操作的执行顺序，避免乱序访问导致的数据错误。
使用同步原语保证关键代码段的串行执行。
实现happens-before关系，确保程序语义的正确性。
02.Python内置线程安全数据结构
a.queue模块详解
a.功能说明
queue模块提供了FIFO队列、LIFO队列和优先级队列等多种线程安全队列。
所有队列操作都是原子性的，内置锁机制保证线程安全。
支持阻塞和非阻塞操作，提供丰富的超时和错误处理机制。
适用于生产者-消费者模式、任务调度、结果缓存等场景。
b.代码示例
---
# queue模块完整使用示例
import threading
import time
import random
import queue
from typing import Any, List, Dict
from collections import namedtuple
import logging
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# 定义数据结构
Task = namedtuple('Task', ['id', 'data', 'priority'])
class QueueSystem:
"""队列系统管理器"""
def __init__(self):
# FIFO队列（先进先出）
self.fifo_queue = queue.Queue(maxsize=100)
# LIFO队列（后进先出）
self.lifo_queue = queue.LifoQueue(maxsize=100)
# 优先级队列（基于优先级排序）
self.priority_queue = queue.PriorityQueue(maxsize=100)
# 统计信息
self.stats = {
'total_tasks': 0,
'completed_tasks': 0,
'failed_tasks': 0,
'queue_sizes': {
'fifo': 0,
'lifo': 0,
'priority': 0
}
}
# 线程池
self.workers = []
self.is_running = False
logger.info("队列系统初始化完成")
def start_workers(self, num_workers: int = 3):
"""启动工作线程"""
self.is_running = True
for i in range(num_workers):
worker = threading.Thread(
target=self._worker_loop,
name=f"Worker-{i+1}",
daemon=True
)
worker.start()
self.workers.append(worker)
logger.info(f"启动了 {num_workers} 个工作线程")
def stop_workers(self):
"""停止工作线程"""
self.is_running = False
# 等待所有工作线程结束
for worker in self.workers:
worker.join(timeout=2.0)
self.workers.clear()
logger.info("所有工作线程已停止")
def add_fifo_task(self, data: Any) -> bool:
"""添加FIFO任务"""
try:
task = Task(
id=self.stats['total_tasks'],
data=data,
priority=0
)
self.fifo_queue.put(task, timeout=1.0)
self.stats['total_tasks'] += 1
return True
except queue.Full:
logger.warning("FIFO队列已满")
return False
def add_lifo_task(self, data: Any) -> bool:
"""添加LIFO任务"""
try:
task = Task(
id=self.stats['total_tasks'],
data=data,
priority=0
)
self.lifo_queue.put(task, timeout=1.0)
self.stats['total_tasks'] += 1
return True
except queue.Full:
logger.warning("LIFO队列已满")
return False
def add_priority_task(self, data: Any, priority: int) -> bool:
"""添加优先级任务"""
try:
task = Task(
id=self.stats['total_tasks'],
data=data,
priority=priority
)
# PriorityQueue按元组第一个元素排序，优先级数字越小越先执行
self.priority_queue.put((priority, task), timeout=1.0)
self.stats['total_tasks'] += 1
return True
except queue.Full:
logger.warning("优先级队列已满")
return False
def _worker_loop(self):
"""工作线程主循环"""
thread_name = threading.current_thread().name
logger.info(f"{thread_name} 开始工作")
while self.is_running:
try:
# 尝试从不同队列获取任务
task = None
queue_type = None
# 首先尝试优先级队列（高优先级任务）
try:
priority_tuple = self.priority_queue.get(timeout=0.1)
task = priority_tuple[1] # 提取Task对象
queue_type = 'priority'
except queue.Empty:
pass
# 如果优先级队列为空，尝试FIFO队列
if task is None:
try:
task = self.fifo_queue.get(timeout=0.1)
queue_type = 'fifo'
except queue.Empty:
pass
# 如果FIFO队列也为空，尝试LIFO队列
if task is None:
try:
task = self.lifo_queue.get(timeout=0.1)
queue_type = 'lifo'
except queue.Empty:
pass
# 如果所有队列都为空，继续等待
if task is None:
continue
# 处理任务
logger.info(f"{thread_name} 从{queue_type}队列获取任务: {task.id}")
self._process_task(task, queue_type)
except Exception as e:
logger.error(f"{thread_name} 处理任务时发生错误: {e}")
self.stats['failed_tasks'] += 1
logger.info(f"{thread_name} 工作结束")
def _process_task(self, task: Task, queue_type: str):
"""处理单个任务"""
try:
start_time = time.time()
# 模拟任务处理
processing_time = random.uniform(0.1, 0.5)
time.sleep(processing_time)
# 任务完成回调
result = {
'task_id': task.id,
'data': task.data,
'processing_time': processing_time,
'thread': threading.current_thread().name,
'queue_type': queue_type,
'completed_at': time.time()
}
logger.info(f"任务 {task.id} 处理完成，耗时 {processing_time:.3f}s")
self.stats['completed_tasks'] += 1
# 可以在这里添加结果回调或通知机制
self._notify_task_completion(result)
except Exception as e:
logger.error(f"处理任务 {task.id} 时发生错误: {e}")
self.stats['failed_tasks'] += 1
def _notify_task_completion(self, result: Dict[str, Any]):
"""任务完成通知"""
# 这里可以实现自定义的完成通知逻辑
pass
def get_queue_info(self) -> Dict[str, Any]:
"""获取队列信息"""
self.stats['queue_sizes'] = {
'fifo': self.fifo_queue.qsize(),
'lifo': self.lifo_queue.qsize(),
'priority': self.priority_queue.qsize()
}
return {
'stats': self.stats.copy(),
'worker_count': len(self.workers),
'is_running': self.is_running
}
class QueueBenchmark:
"""队列性能测试"""
def __init__(self):
self.results = {}
def benchmark_queue_type(self, queue_type: str, num_operations: int = 1000):
"""测试特定队列类型的性能"""
logger.info(f"开始测试 {queue_type} 队列性能")
# 创建相应类型的队列
if queue_type == 'fifo':
test_queue = queue.Queue()
elif queue_type == 'lifo':
test_queue = queue.LifoQueue()
elif queue_type == 'priority':
test_queue = queue.PriorityQueue()
else:
raise ValueError(f"不支持的队列类型: {queue_type}")
# 测试数据
test_data = [f"item_{i}" for i in range(num_operations)]
# 性能测试
start_time = time.time()
# 写入测试
write_start = time.time()
for i, data in enumerate(test_data):
if queue_type == 'priority':
# 优先级队列需要元组格式 (priority, item)
test_queue.put((random.randint(1, 10), data))
else:
test_queue.put(data)
write_time = time.time() - write_start
# 读取测试
read_start = time.time()
items_read = 0
while not test_queue.empty():
item = test_queue.get()
items_read += 1
read_time = time.time() - read_start
total_time = time.time() - start_time
# 记录结果
self.results[queue_type] = {
'operations': num_operations,
'write_time': write_time,
'read_time': read_time,
'total_time': total_time,
'write_ops_per_sec': num_operations / write_time,
'read_ops_per_sec': num_operations / read_time,
'total_ops_per_sec': num_operations * 2 / total_time
}
logger.info(f"{queue_type} 队列测试完成:")
logger.info(f" 写入: {write_time:.3f}s ({self.results[queue_type]['write_ops_per_sec']:.0f} ops/s)")
logger.info(f" 读取: {read_time:.3f}s ({self.results[queue_type]['read_ops_per_sec']:.0f} ops/s)")
logger.info(f" 总计: {total_time:.3f}s ({self.results[queue_type]['total_ops_per_sec']:.0f} ops/s)")
def run_comparison_benchmark(self):
"""运行对比测试"""
queue_types = ['fifo', 'lifo', 'priority']
num_operations = 10000
logger.info(f"开始队列性能对比测试，每种队列 {num_operations} 次操作")
for queue_type in queue_types:
self.benchmark_queue_type(queue_type, num_operations)
# 输出对比结果
logger.info("\n=== 队列性能对比结果 ===")
print(f"{'队列类型':<12} {'写入(ops/s)':<12} {'读取(ops/s)':<12} {'总计(ops/s)':<12}")
print("-" * 50)
for queue_type in queue_types:
result = self.results[queue_type]
print(f"{queue_type:<12} {result['write_ops_per_sec']:<12.0f} "
f"{result['read_ops_per_sec']:<12.0f} {result['total_ops_per_sec']:<12.0f}")
# 使用示例
def main():
"""队列系统使用示例"""
# 1. 基本使用演示
logger.info("=== 基本使用演示 ===")
queue_system = QueueSystem()
# 添加不同类型的任务
queue_system.add_fifo_task("FIFO任务1")
queue_system.add_fifo_task("FIFO任务2")
queue_system.add_lifo_task("LIFO任务1")
queue_system.add_lifo_task("LIFO任务2")
queue_system.add_priority_task("高优先级任务", 1) # 数字越小优先级越高
queue_system.add_priority_task("中优先级任务", 5)
queue_system.add_priority_task("低优先级任务", 10)
# 启动工作线程处理任务
queue_system.start_workers(num_workers=2)
# 让系统运行一段时间处理任务
time.sleep(3)
# 查看队列状态
queue_info = queue_system.get_queue_info()
logger.info(f"队列状态: {queue_info}")
# 添加更多任务测试
for i in range(10):
queue_system.add_fifo_task(f"批量任务_{i}")
time.sleep(2)
# 停止系统
queue_system.stop_workers()
# 2. 性能测试演示
logger.info("\n=== 性能测试演示 ===")
benchmark = QueueBenchmark()
benchmark.run_comparison_benchmark()
if __name__ == "__main__":
main()
---
03.自定义线程安全数据结构
a.线程安全字典实现
a.功能说明
实现一个线程安全的字典类，支持并发读写操作。
使用细粒度锁策略，提高并发性能。
提供原子操作方法，如update_if_absent、compute_if_present等。
支持批量操作和条件更新，减少锁竞争。
b.代码示例
---
import threading
import time
from typing import Any, Dict, Optional, Callable, Iterator
import weakref
import logging
logger = logging.getLogger(__name__)
class ThreadSafeDict:
"""线程安全字典实现"""
def __init__(self, initial_dict: Optional[Dict[Any, Any]] = None):
"""初始化线程安全字典"""
self._data = dict(initial_dict) if initial_dict else {}
self._lock = threading.RLock() # 使用可重入锁
self._stats = {
'gets': 0,
'sets': 0,
'deletes': 0,
'locks_acquired': 0,
'lock_contentions': 0
}
logger.info("ThreadSafeDict 已创建")
def get(self, key: Any, default: Any = None) -> Any:
"""获取值"""
with self._lock:
self._stats['gets'] += 1
return self._data.get(key, default)
def set(self, key: Any, value: Any) -> None:
"""设置值"""
with self._lock:
self._stats['sets'] += 1
self._data[key] = value
def __getitem__(self, key: Any) -> Any:
"""获取值（支持[]语法）"""
with self._lock:
self._stats['gets'] += 1
return self._data[key]
def __setitem__(self, key: Any, value: Any) -> None:
"""设置值（支持[]语法）"""
with self._lock:
self._stats['sets'] += 1
self._data[key] = value
def __delitem__(self, key: Any) -> None:
"""删除键值对"""
with self._lock:
self._stats['deletes'] += 1
del self._data[key]
def __contains__(self, key: Any) -> bool:
"""检查键是否存在"""
with self._lock:
return key in self._data
def __len__(self) -> int:
"""获取字典大小"""
with self._lock:
return len(self._data)
def keys(self) -> list:
"""获取所有键"""
with self._lock:
return list(self._data.keys())
def values(self) -> list:
"""获取所有值"""
with self._lock:
return list(self._data.values())
def items(self) -> list:
"""获取所有键值对"""
with self._lock:
return list(self._data.items())
def update(self, other_dict: Dict[Any, Any]) -> None:
"""批量更新"""
with self._lock:
self._data.update(other_dict)
self._stats['sets'] += len(other_dict)
def clear(self) -> None:
"""清空字典"""
with self._lock:
self._data.clear()
def get_stats(self) -> Dict[str, int]:
"""获取统计信息"""
with self._lock:
return self._stats.copy()
def atomic_update_if_absent(self, key: Any, factory: Callable[[], Any]) -> Any:
"""原子性更新：如果键不存在则创建"""
with self._lock:
if key not in self._data:
self._data[key] = factory()
self._stats['sets'] += 1
return self._data[key]
def compute_if_present(self, key: Any, remapping_function: Callable[[Any], Any]) -> Optional[Any]:
"""如果键存在则重新计算值"""
with self._lock:
if key in self._data:
old_value = self._data[key]
new_value = remapping_function(old_value)
if new_value is None:
del self._data[key]
self._stats['deletes'] += 1
else:
self._data[key] = new_value
self._stats['sets'] += 1
return new_value
return None
class LockFreeCounter:
"""无锁计数器（使用原子操作）"""
def __init__(self, initial_value: int = 0):
self._value = initial_value
self._lock = threading.Lock()
def increment(self) -> int:
"""递增计数器"""
with self._lock:
self._value += 1
return self._value
def decrement(self) -> int:
"""递减计数器"""
with self._lock:
self._value -= 1
return self._value
def get(self) -> int:
"""获取当前值"""
with self._lock:
return self._value
def compare_and_swap(self, expected: int, new_value: int) -> bool:
"""比较并交换操作"""
with self._lock:
if self._value == expected:
self._value = new_value
return True
return False
# 使用示例和性能测试
def test_concurrent_access():
"""测试并发访问性能"""
# 创建线程安全字典
safe_dict = ThreadSafeDict()
# 创建多个线程并发操作字典
def writer_thread(thread_id: int, num_operations: int):
"""写入线程"""
for i in range(num_operations):
key = f"key_{thread_id}_{i}"
value = f"value_{thread_id}_{i}_{time.time()}"
safe_dict.set(key, value)
# 偶尔执行原子操作
if i % 10 == 0:
safe_dict.atomic_update_if_absent(
f"atomic_key_{thread_id}",
lambda: f"atomic_value_{thread_id}_{i}"
)
def reader_thread(thread_id: int, num_operations: int):
"""读取线程"""
for i in range(num_operations):
# 随机读取键
if safe_dict:
keys = safe_dict.keys()
if keys:
key = keys[i % len(keys)]
value = safe_dict.get(key)
# 偶尔执行计算操作
if i % 20 == 0:
safe_dict.compute_if_present(
"test_key",
lambda x: x + "_modified"
)
# 启动测试
num_writers = 5
num_readers = 3
operations_per_thread = 1000
threads = []
# 启动写入线程
for i in range(num_writers):
thread = threading.Thread(
target=writer_thread,
args=(i, operations_per_thread)
)
threads.append(thread)
thread.start()
# 启动读取线程
for i in range(num_readers):
thread = threading.Thread(
target=reader_thread,
args=(i + num_writers, operations_per_thread)
)
threads.append(thread)
thread.start()
# 等待所有线程完成
start_time = time.time()
for thread in threads:
thread.join()
end_time = time.time()
# 输出结果
stats = safe_dict.get_stats()
logger.info(f"并发测试完成:")
logger.info(f" 耗时: {end_time - start_time:.3f}s")
logger.info(f" 字典大小: {len(safe_dict)}")
logger.info(f" 统计: {stats}")
logger.info(f" 总操作数: {sum(stats.values())}")
logger.info(f" 操作速度: {sum(stats.values()) / (end_time - start_time):.0f} ops/s")
def test_counter_performance():
"""测试计数器性能"""
def test_counter(counter, num_operations: int):
"""测试计数器操作"""
start_time = time.time()
for i in range(num_operations):
if i % 2 == 0:
counter.increment()
else:
counter.decrement()
end_time = time.time()
return end_time - start_time
# 测试不同类型的计数器
num_operations = 100000
# 测试带锁计数器
lock_counter = LockFreeCounter()
lock_time = test_counter(lock_counter, num_operations)
logger.info(f"计数器性能测试:")
logger.info(f" 带锁计数器: {lock_time:.3f}s ({num_operations/lock_time:.0f} ops/s)")
logger.info(f" 最终值: {lock_counter.get()}")
if __name__ == "__main__":
test_concurrent_access()
test_counter_performance()
---
现在继续生成第9章的下一个文件： TodoWrite <arg_key>todos</arg_key> <arg_value>[{“content”: “完成第8章 - 08-part4.md (Pipeline模式)”, “status”: “completed”, “activeForm”: “已完成第8章第4小节：Pipeline模式的内容生成”}, {“content”: “完成第8章 - 08-part5.md (Actor模型)”, “status”: “completed”, “activeForm”: “已完成第8章第5小节：Actor模型的内容生成”}, {“content”: “完成第9章 - 09-part0.md (并发安全章节标题)”, “status”: “completed”, “activeForm”: “已完成第9章章节标题文件”}, {“content”: “完成第9章 - 09-part1.md (线程安全的数据结构)”, “status”: “completed”, “activeForm”: “已完成第9章第1小节：线程安全的数据结构”}, {“content”: “完成第9章 - 09-part2.md (死锁检测与避免)”, “status”: “in_progress”, “activeForm”: “正在生成第9章第2小节：死锁检测与避免”}, {“content”: “完成第9章 - 09-part3.md (竞态条件处理)”, “status”: “pending”, “activeForm”: “等待生成第9章第3小节：竞态条件处理”}, {“content”: “完成第9章 - 09-part4.md (原子操作)”, “status”: “pending”, “activeForm”: “等待生成第9章第4小节：原子操作”}, {“content”: “完成第9章 - 09-part5.md (并发测试)”, “status”: “pending”, “activeForm”: “等待生成第9章第5小节：并发测试”}]
9.2 死锁检测与避免
01.死锁概述与成因
a.死锁基本概念
死锁是指两个或多个线程因争夺资源而造成的一种互相等待的现象。
当线程无限期地等待一个永远不会发生的条件时，系统就发生了死锁。
死锁会导致系统资源无法释放，相关线程永远阻塞，影响系统可用性。
了解死锁的成因和预防策略对于编写健壮的并发程序至关重要。
b.死锁产生条件
a.互斥条件
资源不能被多个线程同时使用，一次只能有一个线程占用。
资源一旦被分配给某个线程，就不能被其他线程访问。
这是死锁产生的必要条件，但也正是资源保护的基本要求。
b.占有和等待条件
线程至少持有一个资源，同时等待获取其他线程持有的资源。
线程在等待期间不会释放已占有的资源，形成资源持有链。
这种条件允许死锁链的形成和传播。
c.不可剥夺条件
资源不能被强制性地从持有它的线程中剥夺。
只能等待持有资源的线程自愿释放资源。
缺乏资源抢占机制使得死锁无法被外部打破。
d.循环等待条件
存在一个线程-资源的等待环路，形成循环依赖关系。
线程T1等待T2的资源，T2等待T3的资源，...，Tn等待T1的资源。
环路中的每个线程都在等待其他线程释放资源。
02.死锁预防策略
a.破坏死锁条件
a.破坏互斥条件
对于某些资源，可以考虑设计成可共享访问的模式。
使用读写锁等机制允许多个读者并发访问。
对于独占性要求不强的资源，采用复制或版本化策略。
b.破坏占有和等待条件
要求线程一次性申请所有需要的资源，避免部分持有。
如果无法获取所有资源，则释放已获取的资源并重试。
实现资源申请的原子性操作，确保要么全部成功要么全部失败。
c.破坏不可剥夺条件
当线程申请的资源无法立即获得时，强制释放已持有的资源。
设计资源抢占机制，允许高优先级任务抢占低优先级任务的资源。
实现超时机制，超过等待时间后自动释放资源。
d.破坏循环等待条件
为所有资源分配全局唯一的序号，要求按序号顺序申请资源。
建立资源申请的层次结构，禁止跨层次的资源依赖。
采用统一的资源申请策略，避免形成循环依赖。
03.死锁检测与恢复
a.死锁检测算法
a.资源分配图分析
构建系统的资源分配图，表示线程与资源的分配和等待关系。
使用图论算法检测图中是否存在环路，识别死锁情况。
通过周期性检测和实时监控发现潜在的死锁问题。
b.等待图算法
建立线程间的等待关系图，分析是否存在循环等待。
使用深度优先搜索或拓扑排序检测环路的存在。
实现增量检测机制，提高检测效率。
b.死锁恢复策略
a.线程终止
选择性地终止死锁环中的一个或多个线程。
根据线程优先级、执行成本等因素选择牺牲线程。
被终止线程需要释放所有占有的资源，打破死锁。
b.资源抢占
从死锁线程中强制性地剥夺某些资源。
将剥夺的资源分配给其他等待的线程。
实现回滚机制，允许被抢占线程从安全点重新执行。
04.实践实现
a.死锁检测系统实现
a.功能说明
实现一个完整的死锁检测和预防系统。
提供实时监控和预警机制，及时发现潜在死锁。
支持多种死锁预防策略，可根据场景选择合适的方案。
提供详细的死锁分析和恢复建议。
b.代码示例
---
# 死锁检测与预防系统
import threading
import time
import logging
from typing import Dict, Set, List, Optional, Tuple, Any
from collections import defaultdict, deque
from dataclasses import dataclass, field
from enum import Enum
import weakref
import traceback
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class DeadlockType(Enum):
"""死锁类型"""
CIRCULAR_WAIT = "circular_wait"
RESOURCE_HOLD = "resource_hold"
TIMEOUT_DEADLOCK = "timeout_deadlock"
@dataclass
class ResourceInfo:
"""资源信息"""
resource_id: str
owner_thread: Optional[threading.Thread] = None
waiting_threads: Set[threading.Thread] = field(default_factory=set)
lock_type: str = "normal" # normal, read_write, reentrant
acquire_time: Optional[float] = None
@dataclass
class DeadlockInfo:
"""死锁信息"""
deadlock_type: DeadlockType
involved_threads: Set[threading.Thread]
involved_resources: Set[str]
detection_time: float
wait_graph: Dict[threading.Thread, Set[threading.Thread]] = field(default_factory=dict)
recovery_suggestions: List[str] = field(default_factory=list)
class DeadlockDetector:
"""死锁检测器"""
def __init__(self, check_interval: float = 1.0):
self.resources: Dict[str, ResourceInfo] = {}
self.lock_graph: Dict[threading.Thread, Set[str]] = defaultdict(set)
self.wait_graph: Dict[threading.Thread, Set[threading.Thread]] = defaultdict(set)
self.check_interval = check_interval
self.is_running = False
self.detection_thread: Optional[threading.Thread] = None
# 统计信息
self.stats = {
'detections': 0,
'recovery_actions': 0,
'false_alarms': 0
}
# 回调函数
self.deadlock_callbacks: List[callable] = []
logger.info("死锁检测器已初始化")
def register_resource(self, resource_id: str, lock_type: str = "normal"):
"""注册资源"""
if resource_id not in self.resources:
self.resources[resource_id] = ResourceInfo(
resource_id=resource_id,
lock_type=lock_type
)
logger.debug(f"注册资源: {resource_id}")
def acquire_lock(self, resource_id: str, timeout: Optional[float] = None):
"""获取锁（带死锁检测）"""
current_thread = threading.current_thread()
self.register_resource(resource_id)
resource = self.resources[resource_id]
# 记录等待关系
if resource.owner_thread and resource.owner_thread != current_thread:
self.wait_graph[current_thread].add(resource.owner_thread)
logger.debug(f"线程 {current_thread.name} 等待线程 {resource.owner_thread.name} 的资源 {resource_id}")
# 实时死锁检测
if self._check_immediate_deadlock(current_thread):
deadlock_info = self._analyze_deadlock(current_thread)
logger.warning(f"检测到潜在死锁: {deadlock_info}")
self._handle_deadlock(deadlock_info)
# 尝试获取锁
start_time = time.time()
while True:
if resource.owner_thread is None:
# 获取锁成功
resource.owner_thread = current_thread
resource.acquire_time = time.time()
resource.waiting_threads.discard(current_thread)
self.lock_graph[current_thread].add(resource_id)
# 从等待图中移除
if current_thread in self.wait_graph:
del self.wait_graph[current_thread]
logger.debug(f"线程 {current_thread.name} 获取资源 {resource_id}")
return True
# 检查超时
if timeout and (time.time() - start_time) > timeout:
resource.waiting_threads.discard(current_thread)
logger.warning(f"线程 {current_thread.name} 获取资源 {resource_id} 超时")
return False
# 添加到等待列表
resource.waiting_threads.add(current_thread)
# 短暂等待后重试
time.sleep(0.01)
def release_lock(self, resource_id: str):
"""释放锁"""
current_thread = threading.current_thread()
resource = self.resources.get(resource_id)
if resource and resource.owner_thread == current_thread:
resource.owner_thread = None
resource.acquire_time = None
resource.waiting_threads.clear()
# 从锁图中移除
if current_thread in self.lock_graph:
self.lock_graph[current_thread].discard(resource_id)
logger.debug(f"线程 {current_thread.name} 释放资源 {resource_id}")
def start_monitoring(self):
"""启动死锁监控"""
if not self.is_running:
self.is_running = True
self.detection_thread = threading.Thread(
target=self._monitoring_loop,
name="DeadlockDetector",
daemon=True
)
self.detection_thread.start()
logger.info("死锁监控已启动")
def stop_monitoring(self):
"""停止死锁监控"""
self.is_running = False
if self.detection_thread:
self.detection_thread.join(timeout=2.0)
self.detection_thread = None
logger.info("死锁监控已停止")
def _monitoring_loop(self):
"""监控循环"""
logger.info("死锁检测循环开始运行")
while self.is_running:
try:
# 检测死锁
deadlocks = self._detect_all_deadlocks()
for deadlock in deadlocks:
self.stats['detections'] += 1
logger.warning(f"检测到死锁: {deadlock}")
# 调用回调函数
for callback in self.deadlock_callbacks:
try:
callback(deadlock)
except Exception as e:
logger.error(f"死锁回调函数执行失败: {e}")
# 自动恢复
self._auto_recover_from_deadlock(deadlock)
time.sleep(self.check_interval)
except Exception as e:
logger.error(f"死锁检测循环异常: {e}")
time.sleep(self.check_interval)
logger.info("死锁检测循环已停止")
def _detect_all_deadlocks(self) -> List[DeadlockInfo]:
"""检测所有死锁"""
deadlocks = []
# 构建等待图的快照
wait_graph_snapshot = defaultdict(set)
for thread, waiting_for in self.wait_graph.items():
if thread.is_alive(): # 只考虑活跃线程
wait_graph_snapshot[thread] = {
t for t in waiting_for if t.is_alive()
}
# 检测环路
visited = set()
rec_stack = set()
def dfs_cycle_detection(thread: threading.Thread, path: List[threading.Thread]):
"""深度优先搜索检测环路"""
if thread in rec_stack:
# 找到环路
cycle_start = path.index(thread)
cycle = path[cycle_start:] + [thread]
deadlock = DeadlockInfo(
deadlock_type=DeadlockType.CIRCULAR_WAIT,
involved_threads=set(cycle[:-1]),
involved_resources=set(),
detection_time=time.time(),
wait_graph=dict(wait_graph_snapshot)
)
# 获取涉及的资源
for t in deadlock.involved_threads:
deadlock.involved_resources.update(self.lock_graph.get(t, set()))
deadlocks.append(deadlock)
return
if thread in visited or not thread.is_alive():
return
visited.add(thread)
rec_stack.add(thread)
path.append(thread)
# 访问等待的线程
for waiting_thread in wait_graph_snapshot.get(thread, set()):
dfs_cycle_detection(waiting_thread, path.copy())
rec_stack.remove(thread)
# 对所有活跃线程进行DFS
active_threads = {t for t in threading.enumerate() if t != threading.main_thread()}
for thread in active_threads:
if thread not in visited:
dfs_cycle_detection(thread, [])
return deadlocks
def _check_immediate_deadlock(self, thread: threading.Thread) -> bool:
"""立即检测当前线程的死锁情况"""
# 检查是否存在循环等待
return self._has_cycle_from_thread(thread)
def _has_cycle_from_thread(self, start_thread: threading.Thread) -> bool:
"""检查从指定线程开始是否存在环路"""
visited = set()
rec_stack = set()
def dfs(thread: threading.Thread) -> bool:
if thread == start_thread and start_thread in rec_stack:
return True
if thread in visited or not thread.is_alive():
return False
visited.add(thread)
rec_stack.add(thread)
for waiting_thread in self.wait_graph.get(thread, set()):
if dfs(waiting_thread):
return True
rec_stack.remove(thread)
return False
return dfs(start_thread)
def _analyze_deadlock(self, thread: threading.Thread) -> DeadlockInfo:
"""分析死锁情况"""
involved_threads = set()
involved_resources = set()
# 收集相关线程和资源
def collect_related(t: threading.Thread):
if t in involved_threads or not t.is_alive():
return
involved_threads.add(t)
involved_resources.update(self.lock_graph.get(t, set()))
for waiting_thread in self.wait_graph.get(t, set()):
collect_related(waiting_thread)
collect_related(thread)
return DeadlockInfo(
deadlock_type=DeadlockType.CIRCULAR_WAIT,
involved_threads=involved_threads,
involved_resources=involved_resources,
detection_time=time.time(),
wait_graph=dict(self.wait_graph),
recovery_suggestions=self._generate_recovery_suggestions(involved_threads)
)
def _generate_recovery_suggestions(self, threads: Set[threading.Thread]) -> List[str]:
"""生成恢复建议"""
suggestions = []
# 基于线程优先级建议
thread_priorities = {}
for thread in threads:
# 可以根据线程名称或其他特征确定优先级
priority = 5 # 默认优先级
if "high" in thread.name.lower():
priority = 1
elif "low" in thread.name.lower():
priority = 10
thread_priorities[thread] = priority
# 找出优先级最低的线程
lowest_priority_thread = min(thread_priorities.items(), key=lambda x: x[1])[0]
suggestions.append(f"建议终止线程: {lowest_priority_thread.name} (优先级最低)")
suggestions.append("或者考虑实现超时机制，让线程自动释放资源")
suggestions.append("建议重新设计资源获取顺序，避免循环等待")
return suggestions
def _handle_deadlock(self, deadlock_info: DeadlockInfo):
"""处理死锁"""
logger.warning(f"处理死锁: {deadlock_info.deadlock_type}")
logger.warning(f"涉及线程: {[t.name for t in deadlock_info.involved_threads]}")
logger.warning(f"涉及资源: {deadlock_info.involved_resources}")
# 记录统计
self.stats['detections'] += 1
def _auto_recover_from_deadlock(self, deadlock_info: DeadlockInfo):
"""自动从死锁中恢复"""
try:
# 选择一个线程进行终止
victim_thread = self._select_victim_thread(deadlock_info.involved_threads)
if victim_thread:
logger.warning(f"强制终止线程: {victim_thread.name} 以解决死锁")
# 强制释放线程持有的所有资源
self._force_release_thread_resources(victim_thread)
# 尝试终止线程
# 注意：Python中没有直接终止线程的安全方法
# 这里只是演示，实际应用中需要更复杂的机制
logger.warning(f"已标记线程 {victim_thread.name} 为终止状态")
self.stats['recovery_actions'] += 1
except Exception as e:
logger.error(f"死锁恢复失败: {e}")
def _select_victim_thread(self, threads: Set[threading.Thread]) -> Optional[threading.Thread]:
"""选择牺牲线程"""
if not threads:
return None
# 简单策略：选择持有资源最少的线程
thread_resource_count = {}
for thread in threads:
thread_resource_count[thread] = len(self.lock_graph.get(thread, set()))
return min(thread_resource_count.items(), key=lambda x: x[1])[0]
def _force_release_thread_resources(self, thread: threading.Thread):
"""强制释放线程资源"""
if thread in self.lock_graph:
resources_to_release = list(self.lock_graph[thread])
for resource_id in resources_to_release:
resource = self.resources.get(resource_id)
if resource and resource.owner_thread == thread:
resource.owner_thread = None
resource.acquire_time = None
logger.debug(f"强制释放资源: {resource_id}")
self.lock_graph[thread].clear()
# 从等待图中移除
if thread in self.wait_graph:
del self.wait_graph[thread]
# 移除其他线程对该线程的等待
for waiting_threads in self.wait_graph.values():
waiting_threads.discard(thread)
def add_deadlock_callback(self, callback: callable):
"""添加死锁回调函数"""
self.deadlock_callbacks.append(callback)
def get_statistics(self) -> Dict[str, Any]:
"""获取统计信息"""
return {
'registered_resources': len(self.resources),
'active_locks': sum(1 for r in self.resources.values() if r.owner_thread),
'waiting_threads': len(self.wait_graph),
'stats': self.stats.copy()
}
class DeadlockPreventionManager:
"""死锁预防管理器"""
def __init__(self):
self.resource_hierarchy: Dict[str, int] = {}
self.lock_timeout = 10.0 # 默认锁超时时间
self.max_retry_attempts = 3
def set_resource_hierarchy(self, resources: List[str]):
"""设置资源层次结构"""
for i, resource in enumerate(resources):
self.resource_hierarchy[resource] = i
logger.info(f"资源层次结构已设置: {self.resource_hierarchy}")
def safe_acquire_multiple_locks(self, resources: List[str]) -> bool:
"""安全获取多个锁"""
current_thread = threading.current_thread()
# 按层次顺序排序资源
sorted_resources = sorted(
resources,
key=lambda r: self.resource_hierarchy.get(r, float('inf'))
)
acquired_locks = []
try:
for resource in sorted_resources:
if not self._acquire_lock_with_timeout(resource, self.lock_timeout):
# 获取失败，释放已获取的锁
self._release_multiple_locks(acquired_locks)
return False
acquired_locks.append(resource)
return True
except Exception as e:
logger.error(f"获取多锁失败: {e}")
self._release_multiple_locks(acquired_locks)
return False
def _acquire_lock_with_timeout(self, resource: str, timeout: float) -> bool:
"""带超时获取锁"""
start_time = time.time()
retry_count = 0
while retry_count < self.max_retry_attempts:
try:
# 这里应该调用实际的锁获取逻辑
# 为演示目的，模拟锁获取
logger.debug(f"尝试获取资源: {resource}")
time.sleep(0.01) # 模拟获取时间
return True
except Exception:
retry_count += 1
if time.time() - start_time > timeout:
break
return False
def _release_multiple_locks(self, resources: List[str]):
"""释放多个锁"""
for resource in reversed(resources): # 按相反顺序释放
try:
# 这里应该调用实际的锁释放逻辑
logger.debug(f"释放资源: {resource}")
except Exception as e:
logger.error(f"释放资源 {resource} 失败: {e}")
# 使用示例和测试
def demonstrate_deadlock_detection():
"""演示死锁检测"""
detector = DeadlockDetector(check_interval=0.5)
detector.start_monitoring()
def resource_worker_1():
"""工作线程1 - 可能导致死锁"""
logger.info("线程1 开始工作")
# 获取资源A
detector.acquire_lock("resource_A", timeout=2.0)
time.sleep(1)
# 尝试获取资源B（可能导致死锁）
detector.acquire_lock("resource_B", timeout=5.0)
logger.info("线程1 获取了所有资源")
# 释放资源
detector.release_lock("resource_B")
detector.release_lock("resource_A")
logger.info("线程1 完成工作")
def resource_worker_2():
"""工作线程2 - 可能导致死锁"""
logger.info("线程2 开始工作")
# 获取资源B
detector.acquire_lock("resource_B", timeout=2.0)
time.sleep(1)
# 尝试获取资源A（可能导致死锁）
detector.acquire_lock("resource_A", timeout=5.0)
logger.info("线程2 获取了所有资源")
# 释放资源
detector.release_lock("resource_A")
detector.release_lock("resource_B")
logger.info("线程2 完成工作")
# 启动工作线程
thread1 = threading.Thread(target=resource_worker_1, name="Thread-1")
thread2 = threading.Thread(target=resource_worker_2, name="Thread-2")
thread1.start()
thread2.start()
# 等待一段时间让死锁可能发生
time.sleep(10)
# 停止检测
detector.stop_monitoring()
# 输出统计信息
stats = detector.get_statistics()
logger.info(f"死锁检测统计: {stats}")
def demonstrate_deadlock_prevention():
"""演示死锁预防"""
prevention = DeadlockPreventionManager()
# 设置资源层次
prevention.set_resource_hierarchy(["resource_A", "resource_B", "resource_C"])
def safe_worker():
"""安全工作线程"""
current_thread = threading.current_thread()
logger.info(f"{current_thread.name} 开始安全工作")
# 按层次顺序获取资源
resources = ["resource_A", "resource_B", "resource_C"]
if prevention.safe_acquire_multiple_locks(resources):
try:
logger.info(f"{current_thread.name} 获取了所有资源")
time.sleep(2) # 模拟工作
finally:
# 释放所有资源
prevention._release_multiple_locks(resources)
logger.info(f"{current_thread.name} 释放了所有资源")
else:
logger.warning(f"{current_thread.name} 无法获取所有资源")
# 启动多个安全工作线程
threads = []
for i in range(3):
thread = threading.Thread(target=safe_worker, name=f"SafeWorker-{i+1}")
threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
logger.info("所有安全工作线程已完成")
if __name__ == "__main__":
logger.info("开始死锁检测与预防演示")
# 演示死锁检测
demonstrate_deadlock_detection()
# 等待一段时间
time.sleep(2)
# 演示死锁预防
demonstrate_deadlock_prevention()
logger.info("死锁检测与预防演示完成")
---
9.3 竞态条件处理
01.竞态条件概述
a.基本概念
竞态条件是指多个线程或进程同时访问和操作共享资源时，由于操作时序不确定而导致结果不一致的问题。
当并发操作的正确性依赖于操作执行的相对时序时，就存在竞态条件。
竞态条件是并发编程中最常见和最隐蔽的错误之一，难以复现和调试。
正确识别和处理竞态条件是编写安全并发程序的基础。
b.竞态条件类型
a.数据竞态
多个线程同时读写同一个内存位置，且至少有一个写操作。
没有适当的同步机制保证操作的原子性和一致性。
可能导致数据损坏、内存可见性问题和程序崩溃。
b.检查-时序-执行竞态
基于检查结果进行条件执行的复合操作被多个线程干扰。
经典的时间检查-时间使用(TOCTOU)问题。
常见于状态检查和资源分配等场景。
c.初始化竞态
多个线程同时尝试初始化同一个资源或对象。
可能导致重复初始化、资源泄漏或状态不一致。
单例模式中的懒加载初始化是典型场景。
02.竞态条件检测
a.静态分析技术
a.代码模式识别
搜索常见的竞态条件代码模式。
识别未保护的共享变量访问。
检测复合操作的原子性缺失。
b.数据流分析
分析程序的数据流和控制流。
识别潜在的并发访问冲突。
追踪共享变量的生命周期和访问模式。
b.动态分析技术
a.运行时监控
在程序运行时监控内存访问模式。
检测异常的并发访问行为。
记录竞态条件发生的上下文信息。
b.压力测试
通过高并发场景触发潜在的竞态条件。
使用随机化技术增加竞态条件暴露概率。
结合断言和验证机制捕获错误状态。
03.竞态条件预防与修复
a.同步机制应用
a.互斥锁保护
使用互斥锁保护关键代码段。
确保复合操作的原子性执行。
避免死锁和优先级反转问题。
b.原子操作
使用原子操作保证单个变量的原子性。
实现无锁数据结构和算法。
提高并发性能和可扩展性。
b.设计模式改进
a.不可变对象
使用不可变对象避免状态修改竞态。
通过值传递而不是引用传递。
简化并发编程的复杂性。
b.线程局部存储
使用线程局部变量避免共享状态。
减少同步开销和竞态风险。
提高程序的可扩展性。
04.实践实现
a.竞态条件检测器实现
a.功能说明
实现一个竞态条件检测和预防工具。
提供静态代码分析和动态运行时检测。
支持竞态条件的自动修复建议。
集成到开发流程中，提供实时反馈。
b.代码示例
---
# 竞态条件检测与预防系统
import threading
import time
import logging
import weakref
import inspect
from typing import Dict, Set, List, Optional, Any, Callable, Union
from dataclasses import dataclass, field
from enum import Enum
from collections import defaultdict
import traceback
import ast
import sys
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class RaceConditionType(Enum):
"""竞态条件类型"""
DATA_RACE = "data_race"
CHECK_THEN_ACT = "check_then_act"
INITIALIZATION = "initialization"
ATOMICITY_VIOLATION = "atomicity_violation"
@dataclass
class RaceConditionInfo:
"""竞态条件信息"""
race_type: RaceConditionType
resource_name: str
thread_ids: Set[int]
locations: List[str] # 代码位置
detection_time: float
stack_traces: List[str] = field(default_factory=list)
severity: str = "medium" # low, medium, high, critical
suggestions: List[str] = field(default_factory=list)
@dataclass
class SharedResourceInfo:
"""共享资源信息"""
name: str
current_value: Any
access_history: List[Dict[str, Any]] = field(default_factory=list)
lock_required: bool = True
is_protected: bool = False
last_access_time: Optional[float] = None
access_count: int = 0
class RaceConditionDetector:
"""竞态条件检测器"""
def __init__(self):
self.shared_resources: Dict[str, SharedResourceInfo] = {}
self.thread_access_map: Dict[int, Set[str]] = defaultdict(set)
self.detected_races: List[RaceConditionInfo] = []
self.protection_map: Dict[str, Any] = {} # 资源到锁的映射
self.is_monitoring = False
self.monitor_thread: Optional[threading.Thread] = None
self.detection_callback: Optional[Callable] = None
# 统计信息
self.stats = {
'total_detections': 0,
'data_races': 0,
'check_then_act': 0,
'initialization_races': 0,
'prevention_actions': 0
}
logger.info("竞态条件检测器已初始化")
def register_resource(self, name: str, initial_value: Any, lock_required: bool = True):
"""注册共享资源"""
if name not in self.shared_resources:
self.shared_resources[name] = SharedResourceInfo(
name=name,
current_value=initial_value,
lock_required=lock_required
)
logger.debug(f"注册共享资源: {name}")
def start_monitoring(self):
"""开始监控"""
if not self.is_monitoring:
self.is_monitoring = True
self.monitor_thread = threading.Thread(
target=self._monitoring_loop,
name="RaceDetector",
daemon=True
)
self.monitor_thread.start()
logger.info("竞态条件监控已启动")
def stop_monitoring(self):
"""停止监控"""
self.is_monitoring = False
if self.monitor_thread:
self.monitor_thread.join(timeout=2.0)
self.monitor_thread = None
logger.info("竞态条件监控已停止")
def protect_resource(self, resource_name: str, lock: Any):
"""为资源添加保护"""
self.protection_map[resource_name] = lock
if resource_name in self.shared_resources:
self.shared_resources[resource_name].is_protected = True
logger.debug(f"资源 {resource_name} 已添加保护")
def safe_access(self, resource_name: str, operation: str, callback: Callable) -> Any:
"""安全访问资源"""
current_thread_id = threading.get_ident()
timestamp = time.time()
# 记录访问信息
access_info = {
'thread_id': current_thread_id,
'operation': operation,
'timestamp': timestamp,
'stack_trace': traceback.format_stack()
}
# 检查竞态条件
race_info = self._check_race_condition(resource_name, access_info)
if race_info:
self._handle_race_condition(race_info)
# 更新访问历史
if resource_name in self.shared_resources:
resource = self.shared_resources[resource_name]
resource.access_history.append(access_info)
resource.last_access_time = timestamp
resource.access_count += 1
# 执行操作
if self.protection_map.get(resource_name):
# 使用保护的锁
lock = self.protection_map[resource_name]
try:
if hasattr(lock, 'acquire'):
lock.acquire()
result = callback()
finally:
if hasattr(lock, 'release'):
lock.release()
else:
# 无保护直接访问
result = callback()
return result
def _check_race_condition(self, resource_name: str, access_info: Dict[str, Any]) -> Optional[RaceConditionInfo]:
"""检查竞态条件"""
resource = self.shared_resources.get(resource_name)
if not resource or not resource.lock_required:
return None
current_thread_id = access_info['thread_id']
operation = access_info['operation']
# 检查并发访问
concurrent_accessors = set()
recent_access_time = time.time() - 0.1 # 100ms内的访问
for access in resource.access_history:
if (access['timestamp'] > recent_access_time and
access['thread_id'] != current_thread_id):
concurrent_accessors.add(access['thread_id'])
if len(concurrent_accessors) > 0:
# 检测到潜在的竞态条件
race_type = self._determine_race_type(operation)
race_info = RaceConditionInfo(
race_type=race_type,
resource_name=resource_name,
thread_ids=concurrent_accessors.union({current_thread_id}),
locations=[self._extract_location(access_info)],
detection_time=time.time(),
stack_traces=[access_info['stack_trace']],
severity=self._assess_severity(race_type, resource_name),
suggestions=self._generate_suggestions(race_type, resource_name)
)
return race_info
return None
def _determine_race_type(self, operation: str) -> RaceConditionType:
"""确定竞态条件类型"""
op_lower = operation.lower()
if 'read' in op_lower or 'write' in op_lower:
return RaceConditionType.DATA_RACE
elif 'check' in op_lower and 'act' in op_lower:
return RaceConditionType.CHECK_THEN_ACT
elif 'init' in op_lower:
return RaceConditionType.INITIALIZATION
else:
return RaceConditionType.ATOMICITY_VIOLATION
def _assess_severity(self, race_type: RaceConditionType, resource_name: str) -> str:
"""评估严重程度"""
if race_type == RaceConditionType.DATA_RACE:
return "high"
elif race_type == RaceConditionType.CHECK_THEN_ACT:
return "medium"
elif race_type == RaceConditionType.INITIALIZATION:
return "critical"
else:
return "low"
def _generate_suggestions(self, race_type: RaceConditionType, resource_name: str) -> List[str]:
"""生成修复建议"""
suggestions = []
if race_type == RaceConditionType.DATA_RACE:
suggestions.append(f"为资源 {resource_name} 添加互斥锁保护")
suggestions.append("考虑使用原子操作替换复合操作")
suggestions.append("重新设计数据结构以减少共享状态")
elif race_type == RaceConditionType.CHECK_THEN_ACT:
suggestions.append("使用锁保护检查-执行操作序列")
suggestions.append("考虑使用原子操作或条件变量")
suggestions.append("重新设计逻辑避免竞态条件")
elif race_type == RaceConditionType.INITIALIZATION:
suggestions.append("使用双重检查锁定模式")
suggestions.append("考虑使用模块级别的初始化")
suggestions.append("使用懒加载的单例模式")
return suggestions
def _extract_location(self, access_info: Dict[str, Any]) -> str:
"""提取代码位置信息"""
if 'stack_trace' in access_info and access_info['stack_trace']:
# 提取最相关的栈帧
for frame in access_info['stack_trace']:
if 'File' in frame and 'line' in frame.lower():
return frame.strip()
return "Unknown location"
def _handle_race_condition(self, race_info: RaceConditionInfo):
"""处理检测到的竞态条件"""
self.detected_races.append(race_info)
self.stats['total_detections'] += 1
# 更新分类统计
if race_info.race_type == RaceConditionType.DATA_RACE:
self.stats['data_races'] += 1
elif race_info.race_type == RaceConditionType.CHECK_THEN_ACT:
self.stats['check_then_act'] += 1
elif race_info.race_type == RaceConditionType.INITIALIZATION:
self.stats['initialization_races'] += 1
logger.warning(f"检测到竞态条件: {race_info.race_type.value} 在资源 {race_info.resource_name}")
logger.warning(f"涉及线程: {race_info.thread_ids}")
logger.warning(f"严重程度: {race_info.severity}")
# 调用回调函数
if self.detection_callback:
try:
self.detection_callback(race_info)
except Exception as e:
logger.error(f"竞态条件回调执行失败: {e}")
# 自动预防措施
self._auto_prevent_race(race_info)
def _auto_prevent_race(self, race_info: RaceConditionInfo):
"""自动预防竞态条件"""
resource_name = race_info.resource_name
if resource_name not in self.protection_map and resource_name in self.shared_resources:
# 自动创建锁
lock = threading.Lock()
self.protect_resource(resource_name, lock)
self.stats['prevention_actions'] += 1
logger.info(f"为资源 {resource_name} 自动添加了锁保护")
def _monitoring_loop(self):
"""监控循环"""
logger.info("竞态条件检测循环开始运行")
while self.is_monitoring:
try:
# 清理过期的访问记录
self._cleanup_old_records()
# 分析访问模式，检测复杂的竞态条件
self._analyze_access_patterns()
time.sleep(0.5) # 每500ms检查一次
except Exception as e:
logger.error(f"竞态条件检测循环异常: {e}")
time.sleep(1.0)
logger.info("竞态条件检测循环已停止")
def _cleanup_old_records(self):
"""清理旧的访问记录"""
current_time = time.time()
cutoff_time = current_time - 10.0 # 10秒前的记录
for resource in self.shared_resources.values():
resource.access_history = [
access for access in resource.access_history
if access['timestamp'] > cutoff_time
]
def _analyze_access_patterns(self):
"""分析访问模式"""
for resource_name, resource in self.shared_resources.items():
if len(resource.access_history) < 2:
continue
# 分析访问频率
recent_accesses = [a for a in resource.access_history
if time.time() - a['timestamp'] < 1.0]
if len(recent_accesses) > 10:
logger.warning(f"资源 {resource_name} 访问频率过高 ({len(recent_accesses)} 次/秒)")
# 分析线程访问模式
thread_access_count = defaultdict(int)
for access in recent_accesses:
thread_access_count[access['thread_id']] += 1
if len(thread_access_count) > 1:
logger.debug(f"资源 {resource_name} 被多线程并发访问")
def set_detection_callback(self, callback: Callable):
"""设置检测回调函数"""
self.detection_callback = callback
def get_statistics(self) -> Dict[str, Any]:
"""获取统计信息"""
return {
'monitored_resources': len(self.shared_resources),
'protected_resources': len(self.protection_map),
'detected_races': len(self.detected_races),
'stats': self.stats.copy()
}
def get_race_report(self) -> List[Dict[str, Any]]:
"""获取竞态条件报告"""
report = []
for race in self.detected_races:
report.append({
'type': race.race_type.value,
'resource': race.resource_name,
'threads': list(race.thread_ids),
'severity': race.severity,
'suggestions': race.suggestions,
'detection_time': race.detection_time
})
return report
class ThreadSafeVariable:
"""线程安全变量包装器"""
def __init__(self, name: str, initial_value: Any, detector: Optional[RaceConditionDetector] = None):
self.name = name
self._value = initial_value
self._lock = threading.RLock()
self.detector = detector
if detector:
detector.register_resource(name, initial_value)
detector.protect_resource(name, self._lock)
def get(self) -> Any:
"""获取值"""
if self.detector:
return self.detector.safe_access(
self.name, "read",
lambda: self._value
)
else:
with self._lock:
return self._value
def set(self, value: Any):
"""设置值"""
if self.detector:
self.detector.safe_access(
self.name, "write",
lambda: setattr(self, '_value', value)
)
else:
with self._lock:
self._value = value
def compare_and_swap(self, expected: Any, new_value: Any) -> bool:
"""比较并交换操作"""
if self.detector:
return self.detector.safe_access(
self.name, "compare_and_swap",
lambda: self._compare_and_swap_impl(expected, new_value)
)
else:
with self._lock:
if self._value == expected:
self._value = new_value
return True
return False
def _compare_and_swap_impl(self, expected: Any, new_value: Any) -> bool:
"""CAS操作的内部实现"""
if self._value == expected:
self._value = new_value
return True
return False
def atomic_update(self, update_func: Callable[[Any], Any]):
"""原子更新"""
if self.detector:
self.detector.safe_access(
self.name, "update",
lambda: self._atomic_update_impl(update_func)
)
else:
with self._lock:
self._value = update_func(self._value)
def _atomic_update_impl(self, update_func: Callable[[Any], Any]):
"""原子更新的内部实现"""
self._value = update_func(self._value)
# 使用示例和测试
def demonstrate_race_condition_detection():
"""演示竞态条件检测"""
detector = RaceConditionDetector()
detector.start_monitoring()
# 注册共享资源
shared_counter = ThreadSafeVariable("counter", 0, detector)
shared_flag = ThreadSafeVariable("flag", False, detector)
def race_condition_thread(thread_id: int, num_operations: int):
"""可能产生竞态条件的工作线程"""
thread_name = f"Worker-{thread_id}"
for i in range(num_operations):
# 模拟竞态条件：检查-执行模式
if not shared_flag.get(): # 检查
time.sleep(0.001) # 模拟延迟，增加竞态条件概率
shared_flag.set(True) # 执行
# 修改共享计数器
current = shared_counter.get()
time.sleep(0.001) # 增加竞态条件概率
shared_counter.set(current + 1)
# 模拟一些工作
time.sleep(0.001)
# 重置标志
shared_flag.set(False)
def atomic_operation_thread(thread_id: int, num_operations: int):
"""使用原子操作的工作线程"""
for i in range(num_operations):
# 使用原子操作更新计数器
shared_counter.atomic_update(lambda x: x + 1)
# 使用比较并交换
shared_counter.compare_and_swap(thread_id * 100, thread_id * 100 + 1)
# 启动可能产生竞态条件的线程
race_threads = []
for i in range(3):
thread = threading.Thread(
target=race_condition_thread,
args=(i, 50),
name=f"RaceThread-{i}"
)
race_threads.append(thread)
thread.start()
# 启动使用原子操作的线程
atomic_threads = []
for i in range(2):
thread = threading.Thread(
target=atomic_operation_thread,
args=(i + 10, 50),
name=f"AtomicThread-{i}"
)
atomic_threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in race_threads + atomic_threads:
thread.join()
# 输出结果
logger.info(f"最终计数器值: {shared_counter.get()}")
logger.info(f"最终标志状态: {shared_flag.get()}")
# 获取检测报告
race_report = detector.get_race_report()
logger.info(f"检测到 {len(race_report)} 个竞态条件")
for i, race in enumerate(race_report):
logger.info(f"竞态条件 {i+1}:")
logger.info(f" 类型: {race['type']}")
logger.info(f" 资源: {race['resource']}")
logger.info(f" 严重程度: {race['severity']}")
logger.info(f" 建议修复: {race['suggestions']}")
# 获取统计信息
stats = detector.get_statistics()
logger.info(f"检测器统计: {stats}")
detector.stop_monitoring()
if __name__ == "__main__":
logger.info("开始竞态条件检测演示")
demonstrate_race_condition_detection()
logger.info("竞态条件检测演示完成")
---
05.竞态条件预防最佳实践
a.代码审查指南
a.识别潜在竞态
检查共享变量的访问模式。
分析复合操作的原子性需求。
评估同步机制的充分性。
b.评估修复方案
选择合适的同步原语。
评估性能影响和可扩展性。
确保修复方案的正确性。
b.设计原则
a.最小化共享状态
使用局部变量代替共享变量。
采用消息传递代替共享内存。
设计无状态的服务组件。
b.明确所有权
为每个资源定义明确的所有者。
使用资源管理器控制访问权限。
实现生命周期管理机制。
9.4 原子操作
01.原子操作概述
a.基本概念
原子操作是指不可分割的操作，在执行过程中不会被其他线程中断或干扰。
原子操作要么完全执行成功，要么完全不执行，不会出现部分执行的中间状态。
原子操作是构建无锁数据结构和并发算法的基础，能够有效避免竞态条件。
通过硬件支持的原子指令或软件实现的同步机制保证操作的原子性。
b.原子性保证
a.硬件原子指令
处理器提供单条指令的原子性保证。
支持原子读-修改-写操作序列。
在多核系统中确保缓存一致性。
b.软件原子操作
通过锁机制模拟原子行为。
使用内存屏障确保操作顺序。
实现高级原子操作的原语。
02.Python原子操作基础
a.threading.Lock原子性
a.互斥锁保证
使用互斥锁保护临界区代码。
确保同一时间只有一个线程执行受保护的代码段。
提供基本的原子操作保证机制。
b.锁的开销
锁操作涉及系统调用，开销较大。
可能引起线程阻塞和上下文切换。
在高并发场景下成为性能瓶颈。
b.原子类型支持
a.整数类型
Python GIL为整数操作提供基本的原子性保证。
单个整数变量的简单读写操作是原子的。
复合操作仍然需要额外的同步机制。
b.复合操作
需要锁保护的读-修改-写操作序列。
检查-执行模式的原子性要求。
条件更新和状态转换的原子性保证。
03.高级原子操作实现
a.比较并交换(CAS)
a.CAS操作原理
原子地比较当前值与期望值，相等则更新为新值。
返回操作是否成功的布尔值。
实现无锁算法的核心原语。
b.CAS应用场景
实现无锁计数器和标志位。
构建无锁数据结构。
实现乐观并发控制。
b.原子更新操作
a.原子递增递减
原子地对数值进行加减操作。
支持指定步长的原子操作。
返回更新前后的数值。
b.原子条件更新
基于当前值条件性地更新为新值。
支持复杂的条件判断逻辑。
避免检查-执行的竞态条件。
04.实践实现
a.原子操作框架实现
a.功能说明
实现一个完整的原子操作框架。
支持多种原子操作类型：CAS、递增、递减、条件更新等。
提供性能监控和统计分析功能。
支持自定义原子操作扩展。
b.代码示例
---
# 原子操作框架完整实现
import threading
import time
import logging
from typing import Any, Optional, Callable, Dict, TypeVar, Generic
from dataclasses import dataclass, field
from enum import Enum
from concurrent.futures import ThreadPoolExecutor
import weakref
import traceback
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
T = TypeVar('T')
class AtomicOperationType(Enum):
"""原子操作类型"""
CAS = "compare_and_swap"
INCREMENT = "increment"
DECREMENT = "decrement"
ADD = "add"
UPDATE_IF = "update_if"
GET_AND_SET = "get_and_set"
GET_AND_INCREMENT = "get_and_increment"
@dataclass
class AtomicOperationStats:
"""原子操作统计"""
total_operations: int = 0
successful_operations: int = 0
failed_operations: int = 0
contention_count: int = 0
avg_execution_time: float = 0.0
operation_counts: Dict[str, int] = field(default_factory=dict)
@dataclass
class AtomicOperationResult:
"""原子操作结果"""
operation_type: str
success: bool
old_value: Any
new_value: Optional[Any]
execution_time: float
thread_id: int
retry_count: int = 0
class AtomicVariable(Generic[T]):
"""原子变量类"""
def __init__(self, initial_value: T, name: str = "unnamed"):
self._value = initial_value
self.name = name
self._lock = threading.RLock()
self._stats = AtomicOperationStats()
self._contention_count = 0
self._last_access_time = time.time()
logger.debug(f"原子变量 '{name}' 已初始化，初始值: {initial_value}")
def get(self) -> T:
"""获取当前值（原子读取）"""
start_time = time.time()
thread_id = threading.get_ident()
try:
with self._lock:
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.operation_counts["get"] = self._stats.operation_counts.get("get", 0) + 1
return self._value
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
def set(self, new_value: T) -> T:
"""设置新值（原子写入）"""
start_time = time.time()
thread_id = threading.get_ident()
try:
with self._lock:
old_value = self._value
self._value = new_value
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["set"] = self._stats.operation_counts.get("set", 0) + 1
return old_value
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
def compare_and_swap(self, expected: T, new_value: T) -> bool:
"""比较并交换操作（CAS）"""
start_time = time.time()
thread_id = threading.get_ident()
retry_count = 0
max_retries = 3
while retry_count < max_retries:
try:
with self._lock:
if self._lock._is_locked(): # 检测锁竞争
self._contention_count += 1
if self._value == expected:
self._value = new_value
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["cas"] = self._stats.operation_counts.get("cas", 0) + 1
return True
else:
self._stats.total_operations += 1
self._stats.failed_operations += 1
self._stats.operation_counts["cas"] = self._stats.operation_counts.get("cas", 0) + 1
return False
except RuntimeError as e:
if "locked" in str(e).lower():
retry_count += 1
self._contention_count += 1
time.sleep(0.001 * retry_count) # 指数退避
continue
else:
raise
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
return False
def get_and_set(self, new_value: T) -> T:
"""获取并设置操作"""
start_time = time.time()
thread_id = threading.get_ident()
retry_count = 0
max_retries = 3
while retry_count < max_retries:
try:
with self._lock:
old_value = self._value
self._value = new_value
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["get_and_set"] = self._stats.operation_counts.get("get_and_set", 0) + 1
return old_value
except RuntimeError as e:
if "locked" in str(e).lower():
retry_count += 1
self._contention_count += 1
time.sleep(0.001 * retry_count)
continue
else:
raise
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
raise RuntimeError("get_and_set operation failed after max retries")
def increment(self, delta: T = 1) -> T:
"""原子递增操作"""
if not isinstance(delta, (int, float)):
raise TypeError("delta must be a number for increment operation")
start_time = time.time()
thread_id = threading.get_ident()
retry_count = 0
max_retries = 3
while retry_count < max_retries:
try:
with self._lock:
old_value = self._value
self._value += delta
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["increment"] = self._stats.operation_counts.get("increment", 0) + 1
return old_value
except RuntimeError as e:
if "locked" in str(e).lower():
retry_count += 1
self._contention_count += 1
time.sleep(0.001 * retry_count)
continue
else:
raise
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
raise RuntimeError("increment operation failed after max retries")
def decrement(self, delta: T = 1) -> T:
"""原子递减操作"""
if not isinstance(delta, (int, float)):
raise TypeError("delta must be a number for decrement operation")
return self.increment(-delta)
def get_and_increment(self, delta: T = 1) -> T:
"""获取并递增操作"""
if not isinstance(delta, (int, float)):
raise TypeError("delta must be a number for get_and_increment operation")
start_time = time.time()
thread_id = threading.get_ident()
retry_count = 0
max_retries = 3
while retry_count < max_retries:
try:
with self._lock:
old_value = self._value
self._value += delta
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["get_and_increment"] = self._stats.operation_counts.get("get_and_increment", 0) + 1
return old_value
except RuntimeError as e:
if "locked" in str(e).lower():
retry_count += 1
self._contention_count += 1
time.sleep(0.001 * retry_count)
continue
else:
raise
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
raise RuntimeError("get_and_increment operation failed after max retries")
def update_if(self, condition: Callable[[T], bool], update_func: Callable[[T], T]) -> Optional[T]:
"""条件更新操作"""
start_time = time.time()
thread_id = threading.get_ident()
retry_count = 0
max_retries = 3
while retry_count < max_retries:
try:
with self._lock:
if condition(self._value):
old_value = self._value
self._value = update_func(self._value)
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["update_if"] = self._stats.operation_counts.get("update_if", 0) + 1
return old_value
else:
self._stats.total_operations += 1
self._stats.operation_counts["update_if"] = self._stats.operation_counts.get("update_if", 0) + 1
return None
except RuntimeError as e:
if "locked" in str(e).lower():
retry_count += 1
self._contention_count += 1
time.sleep(0.001 * retry_count)
continue
else:
raise
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
raise RuntimeError("update_if operation failed after max retries")
def atomic_update(self, update_func: Callable[[T], T]) -> T:
"""原子更新操作"""
start_time = time.time()
thread_id = threading.get_ident()
retry_count = 0
max_retries = 3
while retry_count < max_retries:
try:
with self._lock:
old_value = self._value
self._value = update_func(self._value)
self._last_access_time = start_time
self._stats.total_operations += 1
self._stats.successful_operations += 1
self._stats.operation_counts["atomic_update"] = self._stats.operation_counts.get("atomic_update", 0) + 1
return old_value
except RuntimeError as e:
if "locked" in str(e).lower():
retry_count += 1
self._contention_count += 1
time.sleep(0.001 * retry_count)
continue
else:
raise
finally:
execution_time = time.time() - start_time
self._update_execution_time(execution_time)
raise RuntimeError("atomic_update operation failed after max retries")
def _update_execution_time(self, execution_time: float):
"""更新平均执行时间"""
current_avg = self._stats.avg_execution_time
total_ops = self._stats.total_operations
if total_ops > 0:
self._stats.avg_execution_time = (current_avg * (total_ops - 1) + execution_time) / total_ops
def get_statistics(self) -> AtomicOperationStats:
"""获取统计信息"""
return self._stats
def get_contention_info(self) -> Dict[str, Any]:
"""获取竞争信息"""
return {
'contention_count': self._contention_count,
'contention_rate': self._contention_count / max(1, self._stats.total_operations),
'last_access_time': self._last_access_time,
'current_value': self._value
}
class AtomicOperationFramework:
"""原子操作框架"""
def __init__(self):
self.atomic_variables: Dict[str, AtomicVariable] = {}
self.operation_history: List[AtomicOperationResult] = []
self.performance_metrics: Dict[str, Any] = {}
self.is_monitoring = False
self.monitor_thread: Optional[threading.Thread] = None
logger.info("原子操作框架已初始化")
def create_atomic_variable(self, name: str, initial_value: Any) -> AtomicVariable:
"""创建原子变量"""
if name in self.atomic_variables:
raise ValueError(f"原子变量 '{name}' 已存在")
atomic_var = AtomicVariable(initial_value, name)
self.atomic_variables[name] = atomic_var
logger.info(f"创建原子变量: {name} = {initial_value}")
return atomic_var
def get_atomic_variable(self, name: str) -> Optional[AtomicVariable]:
"""获取原子变量"""
return self.atomic_variables.get(name)
def start_monitoring(self):
"""开始性能监控"""
if not self.is_monitoring:
self.is_monitoring = True
self.monitor_thread = threading.Thread(
target=self._monitoring_loop,
name="AtomicMonitor",
daemon=True
)
self.monitor_thread.start()
logger.info("原子操作性能监控已启动")
def stop_monitoring(self):
"""停止性能监控"""
self.is_monitoring = False
if self.monitor_thread:
self.monitor_thread.join(timeout=2.0)
self.monitor_thread = None
logger.info("原子操作性能监控已停止")
def _monitoring_loop(self):
"""监控循环"""
logger.info("原子操作监控循环开始运行")
while self.is_monitoring:
try:
# 收集所有原子变量的统计信息
self._collect_metrics()
# 分析性能数据
self._analyze_performance()
time.sleep(1.0) # 每秒更新一次
except Exception as e:
logger.error(f"原子操作监控异常: {e}")
time.sleep(1.0)
logger.info("原子操作监控循环已停止")
def _collect_metrics(self):
"""收集性能指标"""
total_stats = AtomicOperationStats()
for name, atomic_var in self.atomic_variables.items():
stats = atomic_var.get_statistics()
contention_info = atomic_var.get_contention_info()
# 累积统计数据
total_stats.total_operations += stats.total_operations
total_stats.successful_operations += stats.successful_operations
total_stats.failed_operations += stats.failed_operations
# 累积操作计数
for op_type, count in stats.operation_counts.items():
total_stats.operation_counts[op_type] = total_stats.operation_counts.get(op_type, 0) + count
self.performance_metrics['total_stats'] = total_stats
self.performance_metrics['timestamp'] = time.time()
def _analyze_performance(self):
"""分析性能数据"""
if not self.performance_metrics:
return
total_stats = self.performance_metrics['total_stats']
if total_stats.total_operations > 0:
success_rate = total_stats.successful_operations / total_stats.total_operations
if success_rate < 0.95:
logger.warning(f"原子操作成功率较低: {success_rate:.2%}")
# 分析热点操作
op_counts = total_stats.operation_counts
if op_counts:
most_common_op = max(op_counts.items(), key=lambda x: x[1])
logger.debug(f"最常用的原子操作: {most_common_op[0]} ({most_common_op[1]} 次)")
# 分析竞争情况
for name, atomic_var in self.atomic_variables.items():
contention_info = atomic_var.get_contention_info()
if contention_info['contention_rate'] > 0.1: # 竞争率超过10%
logger.warning(f"原子变量 '{name}' 竞争率较高: {contention_info['contention_rate']:.2%}")
def benchmark_atomic_operations(self, num_threads: int = 4, operations_per_thread: int = 10000):
"""基准测试原子操作性能"""
logger.info(f"开始原子操作基准测试，{num_threads} 个线程，每线程 {operations_per_thread} 次操作")
# 创建测试用的原子变量
test_var = self.create_atomic_variable("benchmark_counter", 0)
test_flag = self.create_atomic_variable("benchmark_flag", False)
def benchmark_worker(worker_id: int):
"""基准测试工作线程"""
start_time = time.time()
for i in range(operations_per_thread):
# 测试各种原子操作
if i % 4 == 0:
test_var.increment(1)
elif i % 4 == 1:
test_var.compare_and_swap(i % 10, (i + 1) % 10)
elif i % 4 == 2:
test_var.get_and_set(worker_id * 100)
else:
test_var.get_and_increment(1)
# 偶尔测试标志变量
if i % 100 == 0:
test_flag.set(not test_flag.get())
execution_time = time.time() - start_time
ops_per_second = operations_per_thread / execution_time
logger.info(f"工作线程 {worker_id} 完成，执行速度: {ops_per_second:.0f} ops/s")
return ops_per_second
# 启动基准测试
with ThreadPoolExecutor(max_workers=num_threads) as executor:
futures = [executor.submit(benchmark_worker, i) for i in range(num_threads)]
results = [future.result() for future in futures]
# 计算总体统计
avg_speed = sum(results) / len(results)
total_ops = num_threads * operations_per_thread
total_time = total_ops / avg_speed
logger.info(f"基准测试完成:")
logger.info(f" 平均速度: {avg_speed:.0f} ops/s")
logger.info(f" 总操作数: {total_ops}")
logger.info(f" 总耗时: {total_time:.3f}s")
# 输出最终值
logger.info(f"最终计数器值: {test_var.get()}")
logger.info(f"最终标志状态: {test_flag.get()}")
return {
'avg_speed': avg_speed,
'total_ops': total_ops,
'total_time': total_time
}
def generate_performance_report(self) -> Dict[str, Any]:
"""生成性能报告"""
report = {
'timestamp': time.time(),
'atomic_variables': len(self.atomic_variables),
'total_operations': 0,
'success_rate': 0.0,
'avg_execution_time': 0.0,
'operation_breakdown': {},
'contention_analysis': {}
}
if self.performance_metrics.get('total_stats'):
total_stats = self.performance_metrics['total_stats']
report['total_operations'] = total_stats.total_operations
report['success_rate'] = (total_stats.successful_operations /
max(1, total_stats.total_operations))
report['avg_execution_time'] = total_stats.avg_execution_time
report['operation_breakdown'] = total_stats.operation_counts.copy()
# 收集竞争分析
for name, atomic_var in self.atomic_variables.items():
contention_info = atomic_var.get_contention_info()
report['contention_analysis'][name] = contention_info
return report
# 使用示例和测试
def demonstrate_atomic_operations():
"""演示原子操作"""
logger.info("开始原子操作演示")
# 创建原子操作框架
framework = AtomicOperationFramework()
framework.start_monitoring()
# 创建原子变量
counter = framework.create_atomic_variable("counter", 0)
flag = framework.create_atomic_variable("flag", False)
status = framework.create_atomic_variable("status", "initial")
def atomic_worker(worker_id: int, num_operations: int):
"""原子操作工作线程"""
thread_name = f"AtomicWorker-{worker_id}"
logger.info(f"{thread_name} 开始执行 {num_operations} 次操作")
success_count = 0
for i in range(num_operations):
try:
if i % 5 == 0:
# 原子递增
old_val = counter.increment()
assert old_val == counter.get() - 1
elif i % 5 == 1:
# 比较并交换
if flag.get():
success = flag.compare_and_swap(True, False)
if success:
success_count += 1
elif i % 5 == 2:
# 条件更新
def is_positive(x):
return x > 0
def make_negative(x):
return -x
result = counter.update_if(is_positive, make_negative)
elif i % 5 == 3:
# 原子更新
def double(x):
return x * 2
old_val = counter.atomic_update(double)
else:
# 获取并递增
old_val = counter.get_and_increment(1)
assert old_val == counter.get() - 1
if i % 1000 == 0:
logger.debug(f"{thread_name} 已完成 {i} 次操作，当前计数器: {counter.get()}")
except Exception as e:
logger.error(f"{thread_name} 操作失败: {e}")
logger.info(f"{thread_name} 完成，成功操作 {success_count} 次")
def data_race_worker(worker_id: int, num_operations: int):
"""数据竞态测试工作线程"""
thread_name = f"RaceWorker-{worker_id}"
# 使用非原子变量进行对比
non_atomic_counter = 0
for i in range(num_operations):
# 模拟数据竞态条件
if non_atomic_counter < 1000000: # 简单的检查
time.sleep(0.000001) # 增加竞态条件概率
non_atomic_counter += 1
# 对比使用原子操作
counter.increment()
if i % 10000 == 0:
logger.debug(f"{thread_name} 非原子计数: {non_atomic_counter}, 原子计数: {counter.get()}")
logger.info(f"{thread_name} 完成，非原子计数: {non_atomic_counter}")
# 启动原子操作工作线程
atomic_threads = []
for i in range(3):
thread = threading.Thread(
target=atomic_worker,
args=(i, 5000),
name=f"AtomicWorker-{i}"
)
atomic_threads.append(thread)
thread.start()
# 启动数据竞态测试线程
race_thread = threading.Thread(
target=data_race_worker,
args=(10, 10000),
name="RaceWorker-Test"
)
race_thread.start()
# 等待所有线程完成
for thread in atomic_threads + [race_thread]:
thread.join()
# 输出结果
logger.info(f"原子计数器最终值: {counter.get()}")
logger.info(f"标志最终状态: {flag.get()}")
logger.info(f"状态最终值: {status.get()}")
# 运行基准测试
benchmark_result = framework.benchmark_atomic_operations(num_threads=4, operations_per_thread=5000)
# 生成性能报告
report = framework.generate_performance_report()
logger.info("原子操作性能报告:")
logger.info(f" 总操作数: {report['total_operations']}")
logger.info(f" 成功率: {report['success_rate']:.2%}")
logger.info(f" 平均执行时间: {report['avg_execution_time']:.6f}s")
logger.info(f" 操作分布: {report['operation_breakdown']}")
framework.stop_monitoring()
logger.info("原子操作演示完成")
if __name__ == "__main__":
demonstrate_atomic_operations()
---
05.原子操作优化策略
a.减少锁竞争
a.细粒度锁策略
使用更小的锁保护范围。
实现分段锁或读写锁。
减少临界区的执行时间。
b.无锁算法设计
使用CAS操作实现无锁数据结构。
避免阻塞式同步机制。
提高并发性能和可扩展性。
b.性能优化技巧
a.批量操作
将多个相关操作合并为单个原子操作。
减少锁获取和释放的开销。
提高整体吞吐量。
b.缓存友好设计
考虑缓存行和内存布局。
减少false sharing问题。
优化数据结构访问模式。
9.5 并发测试
01.并发测试概述
a.基本概念
并发测试是确保多线程和多进程程序正确性和性能的关键环节。
并发程序中的问题往往难以复现和调试，需要专门的测试方法。
通过模拟高并发环境，发现潜在的竞态条件、死锁和性能问题。
提供系统级的测试框架，支持多种并发测试模式。
b.测试类型
a.功能测试
验证并发程序的正确性和功能完整性。
确保多线程环境下的数据一致性和操作原子性。
b.性能测试
测量并发系统的吞吐量、延迟和资源利用率。
分析不同并发度下的性能表现和扩展性。
c.压力测试
在极限负载下测试系统的稳定性和健壮性。
发现系统在高并发下的瓶颈和故障点。
02.并发测试框架实现
a.框架设计
a.功能说明
实现一个完整的并发测试框架，支持多种测试类型。
提供自动化的测试执行、结果收集和报告生成。
集成系统监控，实时分析资源使用情况。
支持测试套件管理，便于组织复杂的测试场景。
b.核心组件
---
# 并发测试框架完整实现
import threading
import time
import logging
import random
import queue
import traceback
import gc
import psutil
import statistics
from typing import List, Dict, Any, Callable, Optional, Tuple, Union
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from enum import Enum
import weakref
import os
# 配置日志
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class TestType(Enum):
"""测试类型枚举"""
FUNCTIONAL = "functional"
PERFORMANCE = "performance"
STRESS = "stress"
RACE_CONDITION = "race_condition"
DEADLOCK = "deadlock"
class TestResult(Enum):
"""测试结果枚举"""
PASSED = "passed"
FAILED = "failed"
ERROR = "error"
TIMEOUT = "timeout"
SKIPPED = "skipped"
@dataclass
class TestMetric:
"""测试指标"""
name: str
value: float
unit: str
description: str = ""
@dataclass
class TestCase:
"""测试用例"""
name: str
test_type: TestType
test_func: Callable
timeout: float = 30.0
thread_count: int = 1
iterations: int = 1
expected_result: Any = None
parameters: Dict[str, Any] = field(default_factory=dict)
def __post_init__(self):
self.metrics = []
self.result = TestResult.SKIPPED
self.error_message = None
self.execution_time = 0.0
class ConcurrentTestFramework:
"""并发测试框架"""
def __init__(self, max_workers: int = None):
self.max_workers = max_workers or min(32, (os.cpu_count() or 1) + 4)
self.test_cases = []
self.test_results = []
self.framework_start_time = time.time()
# 监控数据
self.memory_usage_history = []
self.cpu_usage_history = []
# 测试统计
self.test_stats = {
'total': 0,
'passed': 0,
'failed': 0,
'errors': 0,
'timeouts': 0
}
logger.info(f"并发测试框架初始化完成，最大工作线程数: {self.max_workers}")
def create_stress_test(self, name: str, target_func: Callable,
thread_count: int = 10, duration: float = 60.0,
**kwargs) -> TestCase:
"""创建压力测试用例"""
def stress_test_func():
end_time = time.time() + duration
results = []
def worker():
worker_results = []
while time.time() < end_time:
try:
result = target_func(**kwargs)
worker_results.append(result)
except Exception as e:
logger.error(f"压力测试执行错误: {e}")
time.sleep(random.uniform(0.001, 0.01))
return worker_results
with ThreadPoolExecutor(max_workers=thread_count) as executor:
futures = [executor.submit(worker) for _ in range(thread_count)]
for future in as_completed(futures):
try:
results.extend(future.result())
except Exception as e:
logger.error(f"压力测试线程错误: {e}")
return len(results)
return TestCase(
name=name,
test_type=TestType.STRESS,
test_func=stress_test_func,
thread_count=thread_count,
timeout=duration + 30
)
def create_race_condition_test(self, name: str, shared_resource_init: Callable,
concurrent_operations: List[Callable],
thread_count: int = 5, iterations: int = 1000) -> TestCase:
"""创建竞态条件测试用例"""
def race_test_func():
shared_resource = shared_resource_init()
errors = []
def worker(worker_id: int):
worker_errors = []
for i in range(iterations):
try:
operation = random.choice(concurrent_operations)
result = operation(shared_resource, worker_id, i)
except Exception as e:
worker_errors.append(f"Worker {worker_id} iteration {i}: {str(e)}")
return worker_errors
with ThreadPoolExecutor(max_workers=thread_count) as executor:
futures = [executor.submit(worker, i) for i in range(thread_count)]
for future in as_completed(futures):
try:
errors.extend(future.result())
except Exception as e:
errors.append(f"线程执行错误: {str(e)}")
return len(errors) == 0, errors
return TestCase(
name=name,
test_type=TestType.RACE_CONDITION,
test_func=race_test_func,
thread_count=thread_count
)
def create_performance_test(self, name: str, target_func: Callable,
thread_counts: List[int], iterations: int = 10000,
**kwargs) -> TestCase:
"""创建性能测试用例"""
def performance_test_func():
results = {}
for thread_count in thread_counts:
# 预热
for _ in range(100):
target_func(**kwargs)
# 实际测试
start_time = time.time()
with ThreadPoolExecutor(max_workers=thread_count) as executor:
futures = [executor.submit(target_func, **kwargs) for _ in range(iterations)]
for future in as_completed(futures):
future.result()
end_time = time.time()
results[thread_count] = {
'execution_time': end_time - start_time,
'throughput': iterations / (end_time - start_time)
}
return results
return TestCase(
name=name,
test_type=TestType.PERFORMANCE,
test_func=performance_test_func,
thread_count=max(thread_counts)
)
def run_all_tests(self, sequential: bool = False) -> List[TestCase]:
"""运行所有测试用例"""
logger.info(f"开始执行所有测试，共 {len(self.test_cases)} 个测试用例")
# 重置统计
self.test_stats = {
'total': len(self.test_cases),
'passed': 0,
'failed': 0,
'errors': 0,
'timeouts': 0
}
# 启动监控
self._start_monitoring()
try:
if sequential:
for test_case in self.test_cases:
result = self.run_test_case(test_case)
self.test_results.append(result)
self._update_stats(result)
else:
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
future_to_test = {
executor.submit(self.run_test_case, test_case): test_case
for test_case in self.test_cases
}
for future in as_completed(future_to_test):
try:
result = future.result()
self.test_results.append(result)
self._update_stats(result)
except Exception as e:
test_case = future_to_test[future]
logger.error(f"测试执行异常: {test_case.name} - {e}")
finally:
self._stop_monitoring()
logger.info(f"所有测试执行完成，通过: {self.test_stats['passed']}")
return self.test_results
def run_test_case(self, test_case: TestCase) -> TestCase:
"""运行单个测试用例"""
logger.info(f"开始执行测试: {test_case.name}")
test_case.execution_time = time.time()
try:
if test_case.thread_count > 1:
result = self._run_concurrent_test(test_case)
else:
result = test_case.test_func()
test_case.result = TestResult.PASSED
logger.info(f"测试完成: {test_case.name} - {test_case.result.value}")
except Exception as e:
test_case.result = TestResult.ERROR
test_case.error_message = str(e)
logger.error(f"测试错误: {test_case.name} - {e}")
finally:
test_case.execution_time = time.time() - test_case.execution_time
return test_case
def _run_concurrent_test(self, test_case: TestCase) -> Any:
"""运行并发测试"""
with ThreadPoolExecutor(max_workers=test_case.thread_count) as executor:
future = executor.submit(test_case.test_func)
return future.result(timeout=test_case.timeout)
def _start_monitoring(self):
"""启动系统监控"""
self.monitoring = True
def monitor():
process = psutil.Process()
while self.monitoring:
try:
memory_info = process.memory_info()
cpu_percent = process.cpu_percent()
self.memory_usage_history.append(memory_info.rss / 1024 / 1024)
self.cpu_usage_history.append(cpu_percent)
time.sleep(0.1)
except Exception:
break
self.monitor_thread = threading.Thread(target=monitor, daemon=True)
self.monitor_thread.start()
def _stop_monitoring(self):
"""停止系统监控"""
self.monitoring = False
def _update_stats(self, test_case: TestCase):
"""更新测试统计"""
self.test_stats[test_case.result.value] += 1
class ConcurrentTestSuite:
"""并发测试套件"""
def __init__(self, name: str):
self.name = name
self.framework = ConcurrentTestFramework()
self.suite_start_time = time.time()
def add_functional_test(self, name: str, test_func: Callable,
expected_result: Any = None, **kwargs):
"""添加功能测试"""
test_case = TestCase(
name=f"{self.name}_functional_{name}",
test_type=TestType.FUNCTIONAL,
test_func=test_func,
expected_result=expected_result,
parameters=kwargs
)
self.framework.test_cases.append(test_case)
def add_performance_test(self, name: str, target_func: Callable,
thread_counts: List[int] = None, **kwargs):
"""添加性能测试"""
test_case = self.framework.create_performance_test(
name=f"{self.name}_performance_{name}",
target_func=target_func,
thread_counts=thread_counts or [1, 2, 4, 8],
**kwargs
)
self.framework.test_cases.append(test_case)
def run_all_tests(self, sequential: bool = False) -> Dict[str, Any]:
"""运行套件中的所有测试"""
results = self.framework.run_all_tests(sequential)
return {
'suite_name': self.name,
'suite_execution_time': time.time() - self.suite_start_time,
'test_results': results,
'statistics': self.framework.test_stats
}
---
03.测试用例设计
a.竞态条件测试
a.测试原理
通过多线程同时访问共享资源，检测数据不一致问题。
使用随机化策略增加竞态条件暴露的概率。
b.实现方法
创建共享计数器或数据结构。
多个线程并发执行读-修改-写操作。
验证最终结果的正确性和一致性。
b.死锁检测测试
a.测试场景
模拟循环等待的资源获取模式。
测试锁的超时机制和死锁检测能力。
b.检测机制
监控线程的等待状态和资源持有时间。
使用图算法检测循环等待依赖。
c.性能基准测试
a.测试指标
吞吐量：单位时间内处理的操作数量。
延迟：单个操作的平均执行时间。
资源利用率：CPU和内存的使用效率。
b.测试策略
在不同并发度下进行基准测试。
分析性能随线程数量的变化趋势。
04.并发测试最佳实践
a.测试设计原则
a.可重复性
确保测试结果的可重现性。
使用固定的随机种子和测试参数。
b.可扩展性
设计可配置的测试参数。
支持不同规模的测试场景。
b.测试执行策略
a.渐进式测试
从单线程开始，逐步增加并发度。
在每个阶段验证功能正确性。
b.持续监控
实时监控系统资源使用情况。
及时发现性能瓶颈和异常。
05.并发测试应用案例
a.实际应用场景
a.Web服务器测试
模拟大量并发用户访问。
测试请求处理能力和响应时间。
b.数据库连接池测试
测试连接池的并发性能。
验证连接泄漏和超时处理。
b.案例分析
a.性能问题定位
通过测试发现系统瓶颈。
分析性能数据，提出优化建议。
b.并发问题修复
识别竞态条件和死锁问题。
实施修复方案并验证效果。