Pandas is very effective in scraping html tables and converting them directly to the dataframes. This way we can easily scrape Hang Seng index components from Yahoo! Finance without calling their API.

In [29]:
import pandas as pd

# There there is only one html table on this Yahoo! Finance page
payload=pd.read_html('https://finance.yahoo.com/quote/%5EHSI/components/')

table_0 = payload[0]
df = table_0
In [30]:
df.head()
Out[30]:
Symbol Company Name Last Price Change % Change Volume
0 0006.HK Power Assets Holdings Limited 57.55 0.0 0.00% 695554
1 0003.HK The Hong Kong and China Gas Company Limited 15.50 0.0 0.00% 7774365
2 0386.HK China Petroleum & Chemical Corporation 4.48 0.0 0.00% 39730234
3 0083.HK Sino Land Company Limited 11.22 0.0 0.00% 2887834
4 0700.HK Tencent Holdings Limited 385.80 0.2 +0.05% 8050690

Tickers:

In [31]:
symbols = df['Symbol'].values.tolist()
print(symbols[:10])  # first few tickers

['0006.HK', '0003.HK', '0386.HK', '0083.HK', '0700.HK', '0267.HK', '0016.HK', '0017.HK', '1299.HK', '2628.HK']

Company names:

In [32]:
names = df['Company Name'].values.tolist()
print(names[:10])    # first few company names

['Power Assets Holdings Limited', 'The Hong Kong and China Gas Company Limited', 'China Petroleum & Chemical Corporation', 'Sino Land Company Limited', 'Tencent Holdings Limited', 'CITIC Limited', 'Sun Hung Kai Properties Limited', 'New World Development Company Limited', 'AIA Group Limited', 'China Life Insurance Company Limited']

All companies in Hang Seng index:

In [33]:
df
Out[33]:
Symbol Company Name Last Price Change % Change Volume
0 0006.HK Power Assets Holdings Limited 57.55 0.00 0.00% 695554
1 0003.HK The Hong Kong and China Gas Company Limited 15.50 0.00 0.00% 7774365
2 0386.HK China Petroleum & Chemical Corporation 4.48 0.00 0.00% 39730234
3 0083.HK Sino Land Company Limited 11.22 0.00 0.00% 2887834
4 0700.HK Tencent Holdings Limited 385.80 0.20 +0.05% 8050690
5 0267.HK CITIC Limited 9.66 0.01 +0.10% 2552524
6 0016.HK Sun Hung Kai Properties Limited 115.70 -0.20 -0.17% 1987335
7 0017.HK New World Development Company Limited 10.56 -0.02 -0.19% 6121991
8 1299.HK AIA Group Limited 83.00 0.20 +0.24% 9317851
9 2628.HK China Life Insurance Company Limited 20.60 -0.05 -0.24% 15030055
10 0012.HK Henderson Land Development Company Limited 37.05 -0.10 -0.27% 1984689
11 0883.HK CNOOC Limited 13.04 0.04 +0.31% 26568313
12 1398.HK Industrial and Commercial Bank of China Limited 5.62 -0.02 -0.35% 53451890
13 0011.HK Hang Seng Bank Limited 165.50 -0.60 -0.36% 838406
14 2388.HK BOC Hong Kong (Holdings) Limited 27.35 -0.10 -0.36% 2741255
15 0002.HK CLP Holdings Limited 82.60 0.30 +0.36% 1391934
16 2318.HK Ping An Insurance (Group) Company of China, Ltd. 94.85 0.35 +0.37% 10134877
17 1038.HK CK Infrastructure Holdings Limited 56.00 0.25 +0.45% 581055
18 1997.HK Wharf Real Estate Investment Company Limited 44.65 0.20 +0.45% 1284174
19 2319.HK China Mengniu Dairy Company Limited 31.55 -0.15 -0.47% 2864443
20 0288.HK WH Group Limited 7.96 -0.05 -0.62% 25908682
21 2018.HK AAC Technologies Holdings Inc. 62.50 0.40 +0.64% 3754865
22 1109.HK China Resources Land Limited 34.20 -0.25 -0.73% 8079938
23 0027.HK Galaxy Entertainment Group Limited 56.05 0.45 +0.81% 9356754
24 1093.HK CSPC Pharmaceutical Group Limited 18.70 -0.22 -1.16% 9500257
25 0669.HK Techtronic Industries Company Limited 66.80 0.80 +1.21% 1236219
26 1928.HK Sands China Ltd. 41.30 0.50 +1.23% 9496724
27 1044.HK Hengan International Group Company Limited 62.10 0.90 +1.47% 2800243
28 0101.HK Hang Lung Properties Limited 17.98 0.28 +1.58% 5987014
29 0941.HK China Mobile Limited 69.70 1.70 +2.50% 23194264

Source:

Some web pages might be blocking pandas web scraping. In such case follow this link: