Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-8674][Manager] Pulsar - Modify the calling method from SDK to HTTP #8941

Merged
merged 59 commits into from
Nov 28, 2023
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
8d3d6ea
[Dependabot](deps): Bump actions/checkout from 2 to 3
dependabot[bot] May 1, 2022
c94f632
[INLONG-8674][Manager] Pulsar - Modify the calling method from SDK to…
haibo-duan Sep 20, 2023
51eeeac
Merge branch 'master' into INLONG-8674
haibo-duan Sep 20, 2023
2f421e0
[INLONG-8674][Manager] add license
haibo-duan Sep 20, 2023
145cff1
[INLONG-8674][Manager] fix format violations
haibo-duan Sep 20, 2023
b83d7d9
[INLONG-8674][Manager] fix format violations PulsarUtilsTest
haibo-duan Sep 20, 2023
3eb4d6d
[INLONG-8674][Manager] fix format violations PulsarUtils PulsarResour…
haibo-duan Sep 20, 2023
297d53e
[INLONG-8674][Manager] fix format violations PulsarUtils PulsarResour…
haibo-duan Sep 20, 2023
aafa196
[INLONG-8674][Manager] fix format violations PulsarUtils PulsarResour…
haibo-duan Sep 20, 2023
89c1df0
[INLONG-8674][Manager] fix format violations PulsarUtils PulsarResour…
haibo-duan Sep 20, 2023
59bb210
[INLONG-8674][Manager] remove the pulsar dependency in the pom
haibo-duan Sep 21, 2023
810367e
[INLONG-8674][Manager] remove the pulsar dependency in the pom
haibo-duan Sep 21, 2023
b82b5c8
[INLONG-8674][Manager] remove the pulsar dependency in the pom
haibo-duan Sep 21, 2023
dfa8ca6
[INLONG-8674][Manager] remove the pulsar dependency in the pom
haibo-duan Sep 21, 2023
d332832
[INLONG-8674][Manager] remove the pulsar dependency in the pom
haibo-duan Sep 21, 2023
e7ecc3c
[INLONG-8674][Manager] resolve conflicts
haibo-duan Sep 28, 2023
bbd3a0f
[INLONG-8674][Manager] resolve conflicts
haibo-duan Sep 28, 2023
6176fe4
[INLONG-8674][Manager] resolve conflicts
haibo-duan Sep 28, 2023
2849bdb
Merge branch 'INLONG-8674' of https://github.com/haibo-duan/inlong in…
haibo-duan Sep 28, 2023
046ec5d
chmod 644 to 755
haibo-duan Oct 8, 2023
0b7d211
fix comment
haibo-duan Oct 17, 2023
3f8a743
fix comment
haibo-duan Oct 17, 2023
747345a
[INLONG-8676][Manager] convert hump to underline
haibo-duan Oct 17, 2023
e426d6c
Merge pull request #1 from haibo-duan/dependabot/github_actions/actio…
haibo-duan Oct 17, 2023
fbccbaf
[INLONG-8676][Manager] add specific description
haibo-duan Oct 23, 2023
3653b38
[INLONG-8676][Manager] fix bug :404 Not Found: "HTTP ERROR 404 Not Fo…
haibo-duan Oct 25, 2023
c139b9d
[INLONG-8676][Manager] fix bug :404 Not Found: "HTTP ERROR 404 Not Fo…
haibo-duan Oct 25, 2023
5b0aeb0
[INLONG-8676][Manager] fix bug :404 Not Found: "HTTP ERROR 404 Not Fo…
haibo-duan Oct 25, 2023
337863d
Merge branch 'master' of https://github.com/haibo-duan/inlong into IN…
haibo-duan Oct 25, 2023
fba29e4
Merge branch 'master' of https://github.com/haibo-duan/inlong
haibo-duan Oct 25, 2023
63d0c01
Merge branch 'apache:master' into INLONG-8674
haibo-duan Oct 25, 2023
bd152c7
Merge branch 'master' into INLONG-8674
haibo-duan Oct 25, 2023
e567b60
[INLONG-8676][Manager] merge master
haibo-duan Oct 25, 2023
9521260
Merge branch 'INLONG-8674' of https://github.com/haibo-duan/inlong in…
haibo-duan Oct 25, 2023
ad804bf
[INLONG-8676][Manager] spotless check
haibo-duan Oct 25, 2023
f649ee1
[INLONG-8676][Manager] add test case
haibo-duan Oct 25, 2023
7d90feb
[INLONG-8676][Manager] check style
haibo-duan Oct 25, 2023
2c67d83
[INLONG-8676][Manager] modify pulsar DEFAULT_SERVICE_URL
haibo-duan Oct 25, 2023
3ed556b
[INLONG-8676][Manager] modify pulsar DEFAULT_SERVICE_URL
haibo-duan Oct 25, 2023
c99ed53
[INLONG-8676][Manager] fix bug and add test case
haibo-duan Oct 29, 2023
2e9a13a
[INLONG-8676][Manager] fix bug and add test case
haibo-duan Oct 29, 2023
c5ec819
[INLONG-8676][Manager] mvn spotless:apply to fix these violations
haibo-duan Oct 29, 2023
6a7dbc2
[INLONG-8676][Manager] mvn spotless:apply to fix these violations
haibo-duan Oct 29, 2023
a331d22
[INLONG-8676][Manager] Override the examineMessage method
haibo-duan Oct 31, 2023
c6f3a5b
[INLONG-8676][Manager] mvn spotless:apply
haibo-duan Oct 31, 2023
7507db9
[INLONG-8676][Manager] mvn spotless:apply
haibo-duan Oct 31, 2023
ad5345f
[INLONG-8676][Manager] Add netty configuration in pom file
haibo-duan Nov 1, 2023
375d0c7
[INLONG-8676][Manager] remove netty
haibo-duan Nov 1, 2023
3bba643
[INLONG-8676][Manager] remove netty
haibo-duan Nov 1, 2023
158f152
[INLONG-8676][Manager] fix comments
haibo-duan Nov 2, 2023
f84c5ce
[INLONG-8676][Manager] fix comments
haibo-duan Nov 2, 2023
f6bd178
[INLONG-8676][Manager] fix comments
haibo-duan Nov 2, 2023
360682b
[INLONG-8676][Manager] fix comments
haibo-duan Nov 2, 2023
02bea80
[INLONG-8676][Manager] add sleep time
haibo-duan Nov 2, 2023
a4cf30d
[INLONG-8676][Manager] add sleep time
haibo-duan Nov 5, 2023
f8db827
[INLONG-8676][Manager] Modify constant name
haibo-duan Nov 5, 2023
1310ac0
[INLONG-8676][Manager] Modify variable name.
haibo-duan Nov 13, 2023
c0234b5
Merge branch 'master' of https://github.com/apache/inlong
haibo-duan Nov 17, 2023
c44ede9
Merge branch 'master' into INLONG-8674
haibo-duan Nov 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,19 @@ public static <T> T request(RestTemplate restTemplate, String url, HttpMethod ht
return response.getBody();
}

/**
* Send an void HTTP request
*/
public static void request(RestTemplate restTemplate, String url, HttpMethod httpMethod, Object requestBody,
HttpHeaders header) {
log.debug("begin request to {} by request body {}", url, GSON.toJson(requestBody));
HttpEntity<Object> requestEntity = new HttpEntity<>(requestBody, header);
ResponseEntity<String> response = restTemplate.exchange(url, httpMethod, requestEntity, String.class);

log.debug("success request to {}, status code {}", url, response.getStatusCode());
Preconditions.expectTrue(response.getStatusCode().is2xxSuccessful(), "Request failed");
}

/**
* Send GET request to the specified URL.
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarBrokerEntryMetadata {

private long brokerTimestamp;
private long index;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarLookupTopicInfo {

private String brokerUrl;
private String httpUrl;
private String nativeUrl;
private String brokerUrlSsl;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Map;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarMessageInfo {

private String messageId;
private String topic;
private byte[] body;
private transient Map<String, String> properties;
private boolean poolMessage;
private PulsarBrokerEntryMetadata pulsarBrokerEntryMetadata;
private PulsarMessageMetadata pulsarMessageMetadata;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.List;
import java.util.Map;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarMessageMetadata {

private int payloadSize;
private String partitionKey;
private boolean compactedOut;
private long eventTime;
private boolean partitionKeyB64Encoded;
private byte[] orderingKey;
private long sequenceId;
private boolean nullValue;
private boolean nullPartitionKey;
private Map<String, String> properties;
private long publishTime;
private long deliverAtTime;
private int markerType;
private long txnidLeastBits;
private long txnidMostBits;
private long highestSequenceId;
private String uuid;
private int numChunksFromMsg;
private int totalChunkMsgSize;
private int chunkId;
private String producerName;
private String replicatedFrom;
private int uncompressedSize;
private int numMessagesInBatch;
private String encryptionAlgo;
private String compression;
private byte[] encryptionParam;
private byte[] schemaVersion;
private List<String> replicateTos;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarNamespacePolicies {

private int messageTtlInSeconds;
private PulsarRetentionPolicies retentionPolicies;
private PulsarPersistencePolicies persistence;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarPersistencePolicies {

private int bookkeeperEnsemble;
private int bookkeeperWriteQuorum;
private int bookkeeperAckQuorum;
private double managedLedgerMaxMarkDeleteRate;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarRetentionPolicies {

private int retentionTimeInMinutes;
private long retentionSizeInMB;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Set;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarTenantInfo {

Set<String> adminRoles;

Set<String> allowedClusters;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.inlong.manager.pojo.queue.pulsar;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Map;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PulsarTopicMetadata {

private int partitions;

private boolean deleted;

private Map<String, String> properties;
}
Loading