C ++如何将从Postgresql查询的数据分配给结构并快速存储在std :: vector中

如何解决C ++如何将从Postgresql查询的数据分配给结构并快速存储在std :: vector中

我在postgresql中有大数据,超过1,000,000行。postgresql中的数据结构如下:

CREATE TABLE tickdata
(
    datetime character varying COLLATE pg_catalog."default" NOT NULL,date character varying COLLATE pg_catalog."default" NOT NULL,"time" character varying COLLATE pg_catalog."default" NOT NULL,instrumentid character varying COLLATE pg_catalog."default" NOT NULL,presettlementprice double precision NOT NULL,precloseprice double precision NOT NULL,preopeninterest bigint NOT NULL,openprice double precision NOT NULL,lastprice double precision NOT NULL,highprice double precision NOT NULL,lowprice double precision NOT NULL,averageprice double precision NOT NULL,upperlimit double precision NOT NULL,lowerlimit double precision NOT NULL,bidprice1 double precision NOT NULL,bidvolume1 bigint NOT NULL,askprice1 double precision NOT NULL,askvolume1 bigint NOT NULL,volume bigint NOT NULL,turnover double precision NOT NULL,openinterest bigint NOT NULL,lastvolume bigint NOT NULL,lastopeninterest bigint NOT NULL,gatewayname character varying COLLATE pg_catalog."default" NOT NULL,exchangeid character varying COLLATE pg_catalog."default" NOT NULL,CONSTRAINT hc1605_pkey PRIMARY KEY(datetime)
)

C ++中的数据结构如下:

struct TICK {

    std::string datetime;
    std::string date;
    std::string time;
    std::string instrumentID;
    double preSettlementPrice;
    double preClosePrice;
    int preOpenInterest;
    double openPrice;
    double lastPrice;
    double highPrice;
    double lowPrice;
    double averagePrice;
    double upperLimit;
    double lowerLimit;
    double bidPrice1;
    int bidVolume1;
    double askPrice1;
    int askVolume1;
    int volume;
    double turnover;
    int openInterest;
    int lastVolume;
    int lastOpeninterest;
    std::string ExchangeID;

};

我的意思是,postgresql表中的列名与结构的成员相同。

现在我想从postgresql表中查询1、000、000行数据,并将数据分配给具有TICK结构的向量。

我的代码如下:

std::vector<TICK> tickMap;
std::string sqlQuery = "SELECT * FROM tickdata where datetime between 'XXXXX' and 'XXXXX'";
pqxx::connection conn(testTickConn);
try
{
    start = clock();
    pqxx::nontransaction dbQuery(conn);
    pqxx::result rs = (dbQuery.exec(sqlQuery));
    dbQuery.commit();
    finish = clock();
    double duration = (double)(finish - start) / CLOCKS_PER_SEC;
    emit sendLogData(QString::fromStdString("datamanager load " + instrumentID + " " + std::to_string(rs.size()) + "takes " + std::to_string(duration) + " seconds"));
    start = clock();
    for (auto& row : rs) {

        tick.datetime = QDateTime::fromString(QString::fromStdString(row[1].as<std::string>()),"yyyyMMdd hh:mm:ss.z");
        tick.date = row[2].as<std::string>();
        tick.time = row[3].as<std::string>();
        tick.instrumentID = row[4].as<std::string>();
        tick.preSettlementPrice = row[5].as<double>();
        tick.preClosePrice = row[6].as<double>();

        tick.preOpenInterest = row[7].as<int>();
        tick.openPrice = row[8].as<double>();
        tick.lastPrice = row[9].as<double>();
        tick.highPrice = row[10].as<double>();
        tick.lowPrice = row[11].as<double>();
        tick.averagePrice = row[12].as<double>();
        tick.upperLimit = row[13].as<double>();
        tick.lowerLimit = row[14].as<double>();
        tick.bidPrice1 = row[15].as<double>();
        tick.bidVolume1 = row[16].as<int>();
        tick.askPrice1 = row[17].as<double>();
        tick.askVolume1 = row[18].as<int>();
        tick.volume = row[19].as<int>();
        tick.turnover = row[20].as<double>();
        tick.openInterest = row[21].as<int>();
        tick.lastVolume = row[22].as<int>();
        tick.lastOpeninterest = row[23].as<int>();
        tick.ExchangeID = row[24].as<std::string>();


        //then I want to push back the tick to a map. Maybe I can do something else like put an tick envent....
        tickMap['tickdata'].push_back(tick);
        //emit sendTickData(tick);
    }

    conn.close();

    return true;



}
catch (const std::exception& ex)
{
    emit sendLogData(QString::fromStdString(ex.what()));
    return false;

}

我花了4分钟才能完成这项工作。从postgresql查询数据需要10秒,但是for循环需要我近4分钟。

我发现有2个问题,但我不知道如何解决:

  1. 将格式从字符更改为QDateTime格式花费很长时间。

    tick.datetime = QDateTime :: fromString(QString :: fromStdString(row [1] .asstd :: string()),“ yyyyMMdd hh:mm:ss.z”);

两列需要像这样更改格式。但是我需要精确比较日期时间,以及C ++中其他任何时间格式吗?

  1. 将每行数据分配给TICK结构非常耗时。我可以一次将查询结果分配给矢量吗?

有解决方案吗?我想快点。谢谢。

解决方法

感谢@Sam Varshavchik和@Jesper Juhl的帮助。

我启用了优化功能,预先保留了向量,并创建了一个新结构,该结构仅保留int和double的成员进行测试。

struct TEMPTICK {
    double preSettlementPrice;
    double preClosePrice;
    int preOpenInterest;
    double openPrice;
    double lastPrice;
    double highPrice;
    double lowPrice;
    double averagePrice;
    double upperLimit;
    double lowerLimit;
    double bidPrice1;
    int bidVolume1;
    double askPrice1;
    int askVolume1;
    int volume;
    double turnover;
    int openInterest;
    int lastVolume;
    int lastOpeninterest;

};

新代码如下:

std::vector<TEMPTICK> tickMap;
std::string sqlQuery = "SELECT * FROM tickdata where datetime between 'XXXXX' and 'XXXXX'";
pqxx::connection conn(testTickConn);
try
{
    start = clock();
    pqxx::nontransaction dbQuery(conn);
    pqxx::result rs = (dbQuery.exec(sqlQuery));
    dbQuery.commit();
    finish = clock();
    double duration = (double)(finish - start) / CLOCKS_PER_SEC;
    emit sendLogData(QString::fromStdString("datamanager load " + instrumentID + " " + std::to_string(rs.size()) + "takes " + std::to_string(duration) + " seconds"));
    start = clock();
    tickMap.reserve(re.size());
    for (auto& row : rs) {
        tick.preSettlementPrice = row[5].as<double>();
        tick.preClosePrice = row[6].as<double>();

        tick.preOpenInterest = row[7].as<int>();
        tick.openPrice = row[8].as<double>();
        tick.lastPrice = row[9].as<double>();
        tick.highPrice = row[10].as<double>();
        tick.lowPrice = row[11].as<double>();
        tick.averagePrice = row[12].as<double>();
        tick.upperLimit = row[13].as<double>();
        tick.lowerLimit = row[14].as<double>();
        tick.bidPrice1 = row[15].as<double>();
        tick.bidVolume1 = row[16].as<int>();
        tick.askPrice1 = row[17].as<double>();
        tick.askVolume1 = row[18].as<int>();
        tick.volume = row[19].as<int>();
        tick.turnover = row[20].as<double>();
        tick.openInterest = row[21].as<int>();
        tick.lastVolume = row[22].as<int>();
        tick.lastOpeninterest = row[23].as<int>();


        //then I want to push back the tick to a map. Maybe I can do something else like put an tick envent....
        tickMap.push_back(tick);
        //emit sendTickData(tick);
    }

    conn.close();

    return true;



}
catch (const std::exception& ex)
{
    emit sendLogData(QString::fromStdString(ex.what()));
    return false;

}

所以现在结构成员只有int和double。循环1,000,000行需要40秒。正常速度吗?在将pandas与python一起使用以将1,000行从postgresql加载到数据帧之前,花了不到20秒的时间。我认为c ++应该比熊猫快得多。

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。

相关推荐


依赖报错 idea导入项目后依赖报错,解决方案:https://blog.csdn.net/weixin_42420249/article/details/81191861 依赖版本报错:更换其他版本 无法下载依赖可参考:https://blog.csdn.net/weixin_42628809/a
错误1:代码生成器依赖和mybatis依赖冲突 启动项目时报错如下 2021-12-03 13:33:33.927 ERROR 7228 [ main] o.s.b.d.LoggingFailureAnalysisReporter : *************************** APPL
错误1:gradle项目控制台输出为乱码 # 解决方案:https://blog.csdn.net/weixin_43501566/article/details/112482302 # 在gradle-wrapper.properties 添加以下内容 org.gradle.jvmargs=-Df
错误还原:在查询的过程中,传入的workType为0时,该条件不起作用 &lt;select id=&quot;xxx&quot;&gt; SELECT di.id, di.name, di.work_type, di.updated... &lt;where&gt; &lt;if test=&qu
报错如下,gcc版本太低 ^ server.c:5346:31: 错误:‘struct redisServer’没有名为‘server_cpulist’的成员 redisSetCpuAffinity(server.server_cpulist); ^ server.c: 在函数‘hasActiveC
解决方案1 1、改项目中.idea/workspace.xml配置文件,增加dynamic.classpath参数 2、搜索PropertiesComponent,添加如下 &lt;property name=&quot;dynamic.classpath&quot; value=&quot;tru
删除根组件app.vue中的默认代码后报错:Module Error (from ./node_modules/eslint-loader/index.js): 解决方案:关闭ESlint代码检测,在项目根目录创建vue.config.js,在文件中添加 module.exports = { lin
查看spark默认的python版本 [root@master day27]# pyspark /home/software/spark-2.3.4-bin-hadoop2.7/conf/spark-env.sh: line 2: /usr/local/hadoop/bin/hadoop: No s
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
错误1:Request method ‘DELETE‘ not supported 错误还原:controller层有一个接口,访问该接口时报错:Request method ‘DELETE‘ not supported 错误原因:没有接收到前端传入的参数,修改为如下 参考 错误2:cannot r
错误1:启动docker镜像时报错:Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法:重启docker -&gt; systemctl r
错误1:private field ‘xxx‘ is never assigned 按Altʾnter快捷键,选择第2项 参考:https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2:启动时报错,不能找到主启动类 #
报错如下,通过源不能下载,最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
错误1:maven打包报错 错误还原:使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
错误1:服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
错误1:运行项目后报如下错误 解决方案 报错2:Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案:在pom.
参考 错误原因 过滤器或拦截器在生效时,redisTemplate还没有注入 解决方案:在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-